Error: LCM manifest found in the system is currently more than ### days old during VCF 4.x to 5.x upgrade

Products

• VMware SDDC Manager / VCF Installer
• VMware Cloud Foundation

Issue/Introduction

• When attempting a VMware Cloud Foundation (VCF) 4.x to 5.x upgrade, the precheck process fails. The SDDC Manager UI displays the following error: 

  Description: Checks if LCM manifest is out of date.
  Error Description: LCM Manifest found in the system is currently more than ### days old
  Impact: Medium: May perform upgrade without addressing this issue.
  Remediation: Please manually save or update to the latest manifest.

• Review of the LCM logs (/var/log/vmware/vcf/lcm/lcm-debug.log) reveals signature verification failures:

  ERROR [vcf_lcm, ###########, ###] [c.v.e.s.1.b.d.d.DepotBundleDownloadServiceImp1, Scheduled-3] Error verifying signature for manifest file bundle-###.manifest error INVALID_UNTRUSTED_CERTIFICATE
  ERROR [vcf_lcm, ###########, ###]  [c.v.e.s.1.c. s.BundleManifestDownloadScheduler, Scheduled-3] Exception while polling for bundle manifests com.vmware.evo.sddc.lcm.model.depot.exception.BundleInvalidSignatureException: INVALID_UNTRUSTED_CERTIFICATE
  at com.vmware.evo.sddc.lcm.bundle.download.depot.DepotBundleDownloadServiceImpl.checkManifestSignature(DepotBundleDownloadServiceImpl. java:##

Environment

• VMware Cloud Foundation 4.x
• VMware Cloud Foundation 5.x
• VMware SDDC Manager

Cause

This issue occurs because outdated or corrupted manifest and signature files fail verification. The system triggers a failure when it detects that these files exceed the allowed ##-day validity period or utilize untrusted certificates.

Resolution

To resolve this issue, clean up historical manifests and temporarily disable signature validation.

Note : Take the snapshot of SDDC Manager VM before implementing the resolution steps

1. Remove historical manifest files

1. 1. SSH into the SDDC Manager and elevate to root.

   2. Execute the following command to remove old manifest entries from the database:

      1. For VCF versions prior to 5.1: 

         psql -h localhost -U postgres -d lcm -c "DELETE from manifest WHERE sequence_number < (SELECT MAX(sequence_number) FROM manifest)"

      2. For VCF version 5.1 and higher:

         /usr/pgsql/13/bin/psql -h localhost -U postgres -d lcm -c "DELETE from manifest WHERE sequence_number < (SELECT MAX(sequence_number) FROM manifest)"

2. Disable bundle signature validation
   1. SSH to the SDDC Manager as the VCF user and elevate to root.
   2. Open the LCM properties file for editing:

       vi /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties

   3. Press the Insert key to begin editing. Use the arrow keys to navigate to the desired line, and use Backspace or Delete to remove characters.Locate and set the following parameters to false:

      lcm.depot.adapter.enableBundleSignatureValidation=false 
      lcm.depot.adapter.enableSignatureValidation=false

   4. Save the file and exit.
      

      Press Esc, then type :wq! and press Enter to save the changes and close the file.

   5. Restart the LCM service

      systemctl restart lcm

Additional Information

For further details on the procedures mentioned in the resolution, refer to Cleanup historical manifests and Error BundleSignatureVerificationFailed.