"Failed to get images. Could not reach the server", Image Management tab on SDDC Manager UI fails to load

Products

• VMware Cloud Foundation

Issue/Introduction

• The Lifecycle Management (LCM) service fails to initialize on the SDDC Manager appliance. While the SDDC Manager UI is accessible, workflows involving image management or inventory fail.
  
  
• LCM service fails to start with the error "Temporary failure in name resolution" & "syslog.facility_IS_UNDEFINED is not a valid syslog facility string"
  /var/log/vmware/vcf/lcm/lcm.err reports below,

Failed to instantiate [ch.qos.logback.classic.LoggerContext]
Reported exception:
ch.qos.logback.core.LogbackException: Failed to initialize or to run Configurator: ch.qos.logback.classic.util.DefaultJoranConfigurator
Caused by: java.lang.IllegalArgumentException: syslog.facility_IS_UNDEFINED is not a valid syslog facility string
        at ch.qos.logback.core.net.SyslogAppenderBase.facilityStringToint(SyslogAppenderBase.java:176)
        at ch.qos.logback.classic.pattern.SyslogStartConverter.start(SyslogStartConverter.java:47)
.ConverterUtil.startConverters(ConverterUtil.java:37)
                at ch.qos.logback.classic.net.SyslogAppender.buildLayout(SyslogAppender.java:133)
                at ch.qos.logback.core.net.SyslogAppenderBase.start(SyslogAppenderBase.java:81)
                at org.springframework.boot.logging.logback.LogbackLoggingSystem.loadConfiguration(LogbackLoggingSystem.java:248)
        Caused by: java.net.UnknownHostException: <SDDC ShortName>: Temporary failure in name resolution
                at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
                at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934)
                at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543)
Exception in thread "main" java.lang.reflect.InvocationTargetException
Caused by: java.lang.IllegalStateException: java.lang.IllegalStateException: Logback configuration error detected:
ERROR in ch.qos.logback.classic.pattern.SyslogStartConverter@618b19ad - Could not determine local host name java.net.UnknownHostException: <SDDC ShortName>: <SDDC ShortName>: Temporary failure in name resolution
        at org.springframework.boot.context.logging.LoggingApplicationListener.initializeSystem(LoggingApplicationListener.java:347)
        at org.springframework.boot.context.logging.LoggingApplicationListener.initialize(LoggingApplicationListener.java:298)
logging.logback.LogbackLoggingSystem.reinitialize(LogbackLoggingSystem.java:339)
        at org.springframework.boot.logging.AbstractLoggingSystem.initializeWithConventions(AbstractLoggingSystem.java:74)
        at org.springframework.boot.logging.AbstractLoggingSystem.initialize(AbstractLoggingSystem.java:61

• Image Management tab may report "Failed to get images. Could not reach the server, possible reasons could be network connectivity issue or server going down"
  
  
• /var/log/vmware/vcf/domainmanager/domainmanager.log reports below,
  
  YYYY-MM-DD ERROR [vcf_dm,#####] [c.v.e.s.c.s.a.i.ImageManagementServiceAdapterImpl,ForkJoinPool.commonPool-worker-####]  Error in fetching Images from LCM
com.vmware.cloud.foundation.rest.lcm.runtime.ApiException: Bad Gateway 


• /opt/vmware/sddc-support/sos --services-health reports LCM and LCM_DB as RED.
  
  
  
  
• systemctl status lcm shows the service is in activating (auto-restart) loop as below,
  
  
  
  
• File permissions for  application-prod.properties in the path /opt/vmware/vcf/lcm/lcm-app/conf/ shows the owner as "root" instead of "vcf"
  
  

Environment

• VCF 5.x
• VCF 9.x

Cause

The SDDC Manager appliance hostname differs from the OS-level static hostname. Additionally, the ownership of the file `/opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties` is not set to "vcf"

Resolution

To resolve this issue, the hostname must be updated to ensure successful local name resolution by the LCM service.

1. Log in to the SDDC Manager appliance as vcf via SSH

2. Switch the user to root using  "su" and enter root password

3. Check the current hostname configuration:
   
   hostnamectl

4. Update the hostname to the correct one as per the DNS records.
   
hostnamectl set-hostname test.example.com (Replace test.example.com with the actual lowercase FQDN).

5. Validate the ownership of the file at path "application-prod.properties" by running "ls -ltrh" from the directory /opt/vmware/vcf/lcm/lcm-app/conf.
   
   
6. If the ownership of application-prod.properties is not the vcf user, update it using the following command:
   
chown application-prod.properties vcf_lcm:vcf
   
   
7. Restart the LCM service:

systemctl restart lcm
   
   
8. Verify the service status: 
   
systemctl status lcm