Bundles not showing as available to download in the SDDC/VCF Operations UI Logs show productVersionCatalog.json fails to download with error, "INVALID_SIGNATURE"

Products

• VMware SDDC Manager

Issue/Introduction

While attempting to upgrade via SDDC manager or VCF Operations some bundles don't show as available to download.

Looking that the SDDC manager lcm log located at /var/log/vmware/vcf/lcm/lcm-debug.log We see the following error

####-##-##T##:##:##.###+0000 DEBUG [vcf_lcm,6952b64fd9d4d014295dacc9065345df,c618] [c.v.e.s.l.b.d.c.SignatureVerifier,Scheduled-BMD-1] Loading source file for signature verification:product
VersionCatalog.json
####-##-##T##:##:##.###+0000 INFO  [vcf_lcm,6952b64fd9d4d014295dacc9065345df,c618] [c.v.e.s.l.b.d.c.SignatureVerifier,Scheduled-BMD-1] Invalid signature for: productVersionCatalog.json
####-##-##T##:##:##.###+0000 INFO  [vcf_lcm,6952b64fd9d4d014295dacc9065345df,c618] [c.v.e.s.l.b.d.c.SignatureVerifier,Scheduled-BMD-1] Invalid signature for: productVersionCatalog.json
####-##-##T##:##:##.###+0000 ERROR [vcf_lcm,6952b64fd9d4d014295dacc9065345df,c618] [c.v.e.s.l.s.v.i.ProductVersionCatalogValidator,Scheduled-BMD-1] Error verifying signature for pvc file /n
fs/vmware/vcf/nfs-mount/bundle/depot/local/tmpDir/productVersionCatalog.json error INVALID_SIGNATURE
####-##-##T##:##:##.###+0000 ERROR [vcf_lcm,6952b64fd9d4d014295dacc9065345df,c618] [c.v.e.s.l.s.i.BundleManifestDownloadServiceImpl,Scheduled-BMD-1] Error occurred fetching product version
catalog file from depot
com.vmware.evo.sddc.lcm.model.error.LcmException: Error while verifying signature using trusted sig cert file located at /opt/vmware/vcf/lcm/lcm-app/conf/lcm_bundle_trusted_sig.cert for pro
duct version catalog file /nfs/vmware/vcf/nfs-mount/bundle/depot/local/tmpDir/productVersionCatalog.json.
        at com.vmware.evo.sddc.lcm.services.validation.impl.ProductVersionCatalogValidator.validateProductVersionCatalogFiles(ProductVersionCatalogValidator.java:317)
        at com.vmware.evo.sddc.lcm.services.impl.BundleManifestDownloadServiceImpl.downloadProductVersionCatalog(BundleManifestDownloadServiceImpl.java:516)
        at com.vmware.evo.sddc.lcm.services.impl.BundleManifestDownloadServiceImpl.downloadBundleManifestsInternal(BundleManifestDownloadServiceImpl.java:293)
        at com.vmware.evo.sddc.lcm.services.impl.BundleManifestDownloadServiceImpl.downloadBundleManifests(BundleManifestDownloadServiceImpl.java:202)

Environment

VMware Cloud Foundation 5.x

VMware Cloud Foundation 9.x

Cause

This is a known issue with the ProductVersionCatalog.json sequence 32 file.

Resolution

To workaround the issue please do the following

1. SSH into the SDDC Manager with the vcf user and then elevate to root with su.
   
   
2. Delete or move the productVersionCatalog.json and productVersionCatalog.sig
   
      mv /nfs/vmware/vcf/nfs-mount/bundle/depot/local/tmpDir/productVersionCatalog.json /tmp/productVersionCatalog.json
      mv /nfs/vmware/vcf/nfs-mount/bundle/depot/local/tmpDir/productVersionCatalog.sig /tmp/productVersionCatalog.sig
   
   
3. The productVersionCatalog.json should now download and update the productVersionCatalog.json file located at /nfs/vmware/vcf/nfs-mount/metadata