Unable to update BOM components from the SDDC Manager with the error "Update cannot be started now as compatibility file is missing"

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• In the Updates tab of the relevant WLD, the "Update Now" or "Configure Update" button is greyed out.
• The error "Update cannot be started now as compatibility file is missing" is observed in the "Available Updates" section of the Updates Tab.
• Attempting to manually download the VmwareCompatibilityData.json file fails.
• In the /var/log/vmware/vcf/lcm/lcm-debug.log file of the SDDC manager, the following entries may be observed:

YYYY-MM-DDTHH:MIN DEBUG [vcf_lcm,687a720b50527847becbcf895240bee2,9de0] [c.v.e.s.l.b.d.depot.DepotDownloader,Scheduled-1] Getting file size for [/COMP/SDDC_MANAGER_VCF/manifests/bundle-24282.manifest.sig] from URL[https://dl.broadcom.com:443/<token id>/PROD/COMP/SDDC_MANAGER_VCF/manifests/bundle-24282.manifest.sig]
YYYY-MM-DDTHH:MIN DEBUG [vcf_lcm,687a720b50527847becbcf895240bee2,9de0] [c.v.e.s.l.b.d.depot.DepotDownloader,Scheduled-1] Executing HEAD /<token id>/PROD/COMP/SDDC_MANAGER_VCF/manifests/bundle-24282.manifest.sig
YYYY-MM-DDTHH:MIN INFO  [vcf_lcm,687a71f27432bddbca38bbf18e1cdd4e,229b] [o.a.h.c.h.i.c.HttpRequestRetryExec,Scheduled-9] Recoverable I/O exception (java.net.SocketException) caught when processing request to {s}->https://storage.googleapis.com:443
YYYY-MM-DDTHH:MIN ERROR [vcf_lcm,687a71f27432bddbca38bbf18e1cdd4e,229b] [c.v.e.s.l.c.s.VmwareCompatibilityDataDownloadScheduler,Scheduled-9] VVS Compatibility Data update failed
com.vmware.vcf.compatibility.controllers.internal.vvs.exception.VvsException: The VVS api call to https://vvs.esp.vmware.com/v1/products/bundles/type/vcf-lcm-bundle?format=json failed. Response: Connection reset
        at com.vmware.vcf.compatibility.controllers.internal.vvs.rest.client.VvsApiClient.getBundlesFromApi(VvsApiClient.java:146)
        at com.vmware.vcf.compatibility.controllers.internal.vvs.rest.client.VvsApiClient.getAllProductBundleDetails(VvsApiClient.java:88)
        at com.vmware.vcf.compatibility.controllers.internal.vvs.VvsCompatibilityController.getCompatibilityDataFromRemote(VvsCompatibilityController.java:490)
        at com.vmware.vcf.compatibility.validator.internal.CompatibilityControllerAggregator.getCompatibilityDataFromRemote(CompatibilityControllerAggregator.java:1234)
        at com.vmware.vcf.compatibility.validator.internal.CompatibilityControllerAggregator.getCompatibilityDataFromRemote(CompatibilityControllerAggregator.java:1222)
        at com.vmware.vcf.compatibility.validator.internal.service.CompatibilityValidatorServiceImpl.getCompatibilityDataFromRemote(CompatibilityValidatorServiceImpl.java:240)
        at com.vmware.vcf.compatibility.client.service.impl.CompatibilityValidatorClientImpl.getCompatibilityDataFromRemote(CompatibilityValidatorClientImpl.java:670)
        at com.vmware.evo.sddc.lcm.services.impl.UpgradeCompatibilityServiceImpl.getCompatibilityDataFromRemote(UpgradeCompatibilityServiceImpl.java:722)
        at jdk.internal.reflect.GeneratedMethodAccessor628.invoke(Unknown Source)

Environment

VMware Cloud Foundation 5.x

Cause

This issue occurs because the compatibility file /nfs/vmware/vcf/nfs-mount/compatibility/VmwareCompatibilityData.json is missing on the SDDC Manager appliance.

• The most likely reason for the missing compatability data is that the SDDC manager was upgraded from a previous version where this directory did not exist.
• VVS compatability data was added in VCF 5.0.0.0.

Resolution

Add the compatibility data into the SDDC manager manually

Steps to download the compatability data:

1. Take an offline snapshot of the SDDC Manager appliance.
2. SSH into the SDDC Manager with the vcf user and elevate to root with su root
3. Create and navigate into the compatability directory:
   • mkdir /nfs/vmware/vcf/nfs-mount/compatibility/
   • cd /nfs/vmware/vcf/nfs-mount/compatibility/
4. Run the following command to download the VmwareCompatibilityData.json file:
   • curl --location 'https://vvs.broadcom.com/v1/products/bundles/type/vcf-lcm-v2-bundle?format=json' --header 'X-Vmw-Esp-ClientId: vcf-lcm' > VmwareCompatibilityData.json
5. Set the correct permissions and ownership to the compatability directory
   
   • chown vcf_lcm:vcf -R /nfs/vmware/vcf/nfs-mount/compatibility
   • chmod 755 -R /nfs/vmware/vcf/nfs-mount/compatibility/
6. Restart the LCM service for the changes to take effect:
   
   • systemctl restart lcm


Note: If the SDDC Manager lacks internet access, run the download command on a different Linux machine with access to vvs.broadcom.com, then transfer/copy the file to the appliance.

Update Depot URLs

• The download URLs should also be updated with the download token so that the old repository URLs are removed from the /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties file.
  • See KB: "Depot connection has issues. Depot Invalid User Credential", SDDC fails to download the updates from online repositories

Additional Information

Public URL list for VCF Products

VCF authenticated downloads configuration update instructions