Unable to update BOM components from SDDC Manager with error "Update cannot be started now as compatibility file is missing"

Products

• VMware SDDC Manager

Issue/Introduction

On Workload domain -> "Updates" tab, the "Update now" button is greyed out and the error is displayed as "Update cannot be started now as compatibility file is missing".

Furthermore, an attempt to download VmwareCompatibilityData.json fails. LCM log snippets read as thus.

/var/log/vmware/vcf/lcm/lcm-debug.log

YYYY-MM-DDTHH:MIN DEBUG [vcf_lcm,687a720b50527847becbcf895240bee2,9de0] [c.v.e.s.l.b.d.depot.DepotDownloader,Scheduled-1] Getting file size for [/COMP/SDDC_MANAGER_VCF/manifests/bundle-24282.manifest.sig] from URL[https://dl.broadcom.com:443/<token id>/PROD/COMP/SDDC_MANAGER_VCF/manifests/bundle-24282.manifest.sig]
YYYY-MM-DDTHH:MIN DEBUG [vcf_lcm,687a720b50527847becbcf895240bee2,9de0] [c.v.e.s.l.b.d.depot.DepotDownloader,Scheduled-1] Executing HEAD /<token id>/PROD/COMP/SDDC_MANAGER_VCF/manifests/bundle-24282.manifest.sig
YYYY-MM-DDTHH:MIN INFO  [vcf_lcm,687a71f27432bddbca38bbf18e1cdd4e,229b] [o.a.h.c.h.i.c.HttpRequestRetryExec,Scheduled-9] Recoverable I/O exception (java.net.SocketException) caught when processing request to {s}->https://storage.googleapis.com:443
YYYY-MM-DDTHH:MIN ERROR [vcf_lcm,687a71f27432bddbca38bbf18e1cdd4e,229b] [c.v.e.s.l.c.s.VmwareCompatibilityDataDownloadScheduler,Scheduled-9] VVS Compatibility Data update failed
com.vmware.vcf.compatibility.controllers.internal.vvs.exception.VvsException: The VVS api call to https://vvs.esp.vmware.com/v1/products/bundles/type/vcf-lcm-bundle?format=json failed. Response: Connection reset
        at com.vmware.vcf.compatibility.controllers.internal.vvs.rest.client.VvsApiClient.getBundlesFromApi(VvsApiClient.java:146)
        at com.vmware.vcf.compatibility.controllers.internal.vvs.rest.client.VvsApiClient.getAllProductBundleDetails(VvsApiClient.java:88)
        at com.vmware.vcf.compatibility.controllers.internal.vvs.VvsCompatibilityController.getCompatibilityDataFromRemote(VvsCompatibilityController.java:490)
        at com.vmware.vcf.compatibility.validator.internal.CompatibilityControllerAggregator.getCompatibilityDataFromRemote(CompatibilityControllerAggregator.java:1234)
        at com.vmware.vcf.compatibility.validator.internal.CompatibilityControllerAggregator.getCompatibilityDataFromRemote(CompatibilityControllerAggregator.java:1222)
        at com.vmware.vcf.compatibility.validator.internal.service.CompatibilityValidatorServiceImpl.getCompatibilityDataFromRemote(CompatibilityValidatorServiceImpl.java:240)
        at com.vmware.vcf.compatibility.client.service.impl.CompatibilityValidatorClientImpl.getCompatibilityDataFromRemote(CompatibilityValidatorClientImpl.java:670)
        at com.vmware.evo.sddc.lcm.services.impl.UpgradeCompatibilityServiceImpl.getCompatibilityDataFromRemote(UpgradeCompatibilityServiceImpl.java:722)
        at jdk.internal.reflect.GeneratedMethodAccessor628.invoke(Unknown Source)

Environment

VMware Cloud Foundation 5.x

Cause

This issue happens as compatibility file "/nfs/vmware/vcf/nfs-mount/compatibility/VmwareCompatibilityData.json" is missing in SDDC Manager appliance.

Resolution

Place the compatibility data in SDDC manager manually.

Steps:

• Take Offline Snapshot of the SDDC Manager VM
• SSH to SDDC Manager with vcf user and su to root
  • mkdir /nfs/vmware/vcf/nfs-mount/compatibility/
  • cd /nfs/vmware/vcf/nfs-mount/compatibility/
  • Download the VmwareCompatibilityData.json file as per the command below (Please run below command on the machine with access to the internet)-
    • curl --location 'https://vvs.esp.vmware.com/v1/products/bundles/type/vcf-lcm-bundle?format=json' --header 'X-Vmw-Esp-ClientId: vcf-lcm' > VmwareCompatibilityData.json
  • Run below commands to give required permission:
    
    • chown vcf_lcm:vcf -R /nfs/vmware/vcf/nfs-mount/compatibility
    • chmod 755 -R /nfs/vmware/vcf/nfs-mount/compatibility/
  • systemctl restart lcm