Lifecycle Metadata Sync Failed on Binary Management tab of Fleet Management

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• Login to VCF Operations > Fleet Management > Select VCF instance > Binary Management, you will see error "Could not reach the server, possible reason could be network connectivity issue or server going down". Even when attempted to "Sync Now" on same page fails. Looking at "Last attempt failed. View details" it will show as below,
  
  
  
• Connect to SDDC manager via SSH and running command "systemctl status lcm" shows (auto-restart) (Result: exit-code) as below,
  
  Active: activating (auto-restart) (Result: exit-code) since ### ####-##-####:##:##UTC; 18s ago
Process: ###### ExecStart=/opt/vmware/vcf/lcm/lcm-app/bin/lcm (code=exited, status=1/FAILURE)
  
  
• /var/log/vmware/vcf/lcm/lcm-debug.log on SDDC manager shows error as below,
  
  Caused by: java.lang.IllegalArgumentException: Could not resolve placeholder 'lcm.depot.adapter.remote.v2.rootDir' in value "${lcm.depot.adapter.remote.v2.rootDir}"
           at org.springframework.util.PropertyPlaceholderHelper.parseStringValue(PropertyPlaceholderHelper.java:169)
           at org.springframework.util.PropertyPlaceholderHelper.replacePlaceholders(PropertyPlaceholderHelper.java:115)
  
  
• cat /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties | grep "rootDir" output may show below line,
  
  lcm.depot.adapter.remote.rootDir=/<download token>/PROD
  
  

Environment

VMware Cloud Foundation (VCF) 9.x

Cause

The lcm.depot.adapter.remote.v2.rootDir parameter has been removed or is missing from /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties file. This occurs if application-prod.properties file was edited in past.

Resolution

To resolve this issue follow through below steps,

1. Take a snapshot of the SDDC manager.
   
   
2. SSH into the affected SDDC Manager using the vcf user and switch to the root user using su.
   
   
3. Create a backup of the application-prod.properties file as below,
   cp /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties.bak
   
   
4. Edit application-prod.properties,
   vi /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties
   
   
5. Remove the offline depot configuration line as below if it exists, (Note: ignore step.5, if below line is missing)
   ################### LCM DEPOT PROPERTIES ########################
lcm.depot.adapter.host=dl.broadcom.com
lcm.depot.adapter.port=443
lcm.depot.adapter.remote.rootDir=/<download token>/PROD        <--- Line to be removed if exisiting
   
   
6. Add "lcm.depot.adapter.remote.v2.rootDir=/PROD" as shown below,
   ################### LCM DEPOT PROPERTIES ########################
lcm.depot.adapter.host=dl.broadcom.com
lcm.depot.adapter.port=443
lcm.depot.adapter.remote.v2.rootDir=/PROD      <---- Missing Line
   
   
7. Save the file application-prod.properties and exit.
   
   
8. Restart the LCM service on SDDC manager as below,
   systemctl restart lcm