Clearing the bundle cache on VCF 9.1 Fleet Depot Service

Products

• VCF Operations/Automation (formerly VMware Aria Suite)
• VMware SDDC Manager / VCF Installer
• VMware Cloud Foundation

Issue/Introduction

• You are unable to download or delete a stale binary from the Software Depot (FDS) on the VMware Cloud Foundation (VCF) Operations 9.1 Fleet.
  
  
• When attempting to download the binary from the UI by navigating to Build > Lifecycle > VCF Instances > [Specific instance] > Installed > [Select component] > VCF Services runtime > Download, the process fails with the following error message:
  
  "Download failed" - "Your download failed because of an LcmException: Patch file /nfs/vmware/vcf/nfs-mount/bundle/depot/local/bundles/<Path>/<File_name> has a checksum of <id>, which does not match the expected checksum <id>. Please retry the download."
  
  
• The following LcmException is observed in the SDDC Manager /var/log/vmware/vcf/lcm/lcm-debug.log:
  
   ERROR [vcf_lcm,#########] [c.v.e.s.l.s.i.BundleDownloadExceptionHandlerImpl,Bundle-Downloader-2] Bundle download failed
com.vmware.evo.sddc.lcm.model.error.LcmException: Patch file /nfs/vmware/vcf/nfs-mount/bundle/depot/local/bundles/<path>/<Filename> checksum <id> does NOT match with the element checksum <id>
        at com.vmware.evo.sddc.lcm.file.common.FileChecksumUtil.checkFileSizeAndChecksum(FileChecksumUtil.java:###)
    
• Additionally, a cache_status="MISS" or "HIT" entry is observed in the Software Depot (FDS) console.log:
  
  stdout F ###.###.##.## - svcVCFOpsDepot  depot_type="OFFLINE" "GET /depot-service/content-gateway/PROD/metadata/manifest/v1/vcfManifest.json HTTP/1.1" 200 212977 "-" "Apache-HttpClient/5.5.1 (Java/21.0.9)" rt=0.003 uct="0.000" uht="0.001" urt="0.002" cache_status="MISS" cache_key="GET/depot-service/content-gateway/PROD/metadata/manifest/v1/vcfManifest.json" upstream="127.0.0.1:9080" upstream_status="200"
  
  or
  
  stdout F - ##### depot_type="OFFLINE" "GET /depot-service/content-gateway/PROD/COMP/VSP/<file_name> HTTP/1.1" 200 7045831488 "-" "Apache-HttpClient/5.5.1 (Java/21.0.9)" rt=26.015 uct="-" uht="-" urt="-" cache_status="HIT" cache_key="GET/depot-service/content-gateway/PROD/COMP/VSP/<fileName>" upstream="-" upstream_status="-"

Environment

 

• VMware Cloud Foundation (VCF) Operations 9.1

• VMware Cloud Foundation (VCF) 9.1

 

Cause

This issue is caused by potential cache corruption within the local Software Depot. A checksum mismatch occurs because the downloaded binary does not match the expected SHA256 signature, indicating that partial or corrupted data is residing in the local depot cache.

Resolution

To resolve this issue, clear the corrupted cache from the SDDC Manager using the provided cleanup script:

1. Download the cleanup_fds_depot_cache.sh script attached to this KB article.

2. Copy the script to the /home/vcf/ directory on the SDDC Manager.

3. Open an SSH session to the SDDC manager using the vcf user account.

4. Elevate your privileges to root by running the following command:

   su
   

5. Update the script permissions to make it executable:

   chmod +x cleanup_fds_depot_cache.sh
   

6. Run the script and follow the on-screen prompts to clear the cache:

   Usage:

  ./cleanup_fds_depot_cache.sh [OPTIONS]

   Options (can also be set as environment variables):
  --vmsp-node-host  Hostname/IP of the VMSP kubectl node to SSH into (env: SSH_HOST)
                    If omitted, runs locally.
  -h, --help      Show this help message