Managing Automatic Cleanup of Orphaned OCI Images in VCF Software Depot.

Products

VMware SDDC Manager / VCF Installer VMware Cloud Foundation

Issue/Introduction

In VMware Cloud Foundation (VCF) 9.1.1 , automatic deletion of untracked or orphaned OCI images during startup reconciliation in the Fleet Download Service (FDS) is disabled by default (deleteOrphanedImagesEnabled: false) to avoid accidentally deleting valid externally uploaded images.

Because this is the default safe behavior, administrators will not notice immediate operational symptoms under normal conditions. However, the following scenario may occur:

Bundle Download or Import Failure due to Storage Limits: A software bundle download, import, or upload task fails with out-of-disk-space or storage capacity exceeded errors because unused or orphaned OCI image layers in the registry are not automatically deleted.

Environment

VMware Cloud Foundation (VCF) / VMware vSphere Foundation (VVF) 9.1.1
VCF Software Depot (Fleet Depot Server)

Cause

During startup reconciliation, VCF Software Depot scans the embedded OCI registry catalog and compares it against on-disk metadata JSON files (under /data/depot/PROD/COMP//). Any image present in the registry catalog that is not referenced by an on-disk metadata JSON file is identified as an orphaned image.

Operations and Conditions Leading to Orphaned OCI Images:

External / Manual OCI Image Pushes:
OCI images were pushed directly into the VCF Software Depot Docker registry using external CLI tools (e.g., imgpkg) without calling the official VCF Software Depot upload API or generating on-disk metadata JSON files.
Interrupted Bundle Deletion:
A bundle deletion operation removed the metadata JSON file from disk, but the process was interrupted (e.g., pod crash or system restart) before all associated OCI image tags were deleted from the Docker registry.
Because VCF Software Depot cannot programmatically distinguish between a true orphan (interrupted upload/deletion) and a valid OCI image pushed via external tools, automatic deletion defaults to false (deleteOrphanedImagesEnabled: false) to prevent accidental loss of valid customer-uploaded images.

Resolution

How to Check if There are Orphaned OCI Images in Software Depot (Fleet Depot Service)

Please refer to KB: OCI images uploaded manually to VCF Software Depot are not recognized by vcf-download-tool in air-gapped environments

There are steps and script to check if there are orphaned OCI images.

Only set deleteOrphanedImagesEnabled to true when there're currently no orphaned OCI images OR these orphaned OCI images are no longer needed.

How to Enable or Disable Automatic Orphaned OCI Image Deletion

By default, deleteOrphanedImagesEnabled is set to false. If you want VCF Software Depot to automatically delete orphaned OCI images during startup reconciliation to free up storage space, follow the steps below using the VMSP Platform Config API

Caution: Before setting deleteOrphanedImagesEnabled to true, ensure that no supported external OCI images reside in the registry without corresponding on-disk metadata such as uploading via imgpkg following the VKS Deployment Guide for VCF 9.1.0 air-gapped environments. Once enabled, any OCI image in the registry not referenced by metadata JSON files will be permanently deleted upon VCF Software Depot startup reconciliation.

Download this shell script update-fds-oci-cleanup.sh (https://github.com/vmware/vsphere-supervisor/blob/main/airgapped/scripts/update-fds-oci-cleanup.sh) , set your environment variables, and run it with true (to enable) or false (to disable).

Procedure: Enabling or Disabling Automatic Cleanup

Download the attached update-fds-oci-cleanup.sh script.
Identify a Control Plane Node from the VCF Operations UI.

Build > Lifecyle > VCF Management > Components > VCF Services Runtime
Copy the update-fds-oci.cleanup.sh script to the Control Plane node.
SSH into the Control Plane node with the vmware-system-user password. (If password unknown follow steps in, "Resetting the vmware-system-user password for VMware Cloud Foundation Services Runtime")
Elevate to root user with sudo -i
Make the script executable.

chmod +x update-fds-oci-cleanup.sh
Fill the following information and then paste the three commands into the SSH session.

export PLATFORM_HOST=""
export ADMIN_USERNAME="[email protected]"
export ADMIN_PASSWORD=""
Execute the script

./update-fds-oci-cleanup.sh true Example output: ==> 1. Obtaining access token from VMSP Identity Service... ==> 2. Discovering FDS Component ID... Found FDS Component ID: bb5763bb-f7cd-4564-8307-a9370a75a138 ==> 3. Submitting configuration update (deleteOrphanedImagesEnabled=true)... Task submitted successfully. Task ID: aze6qw7gxvdi3ntrq7t6tydkru ==> 4. Polling task status (Timeout: 20 minutes)... Attempt 1/120 - Task Status: Pending Attempt 2/120 - Task Status: Pending Attempt 3/120 - Task Status: Running ... (additional Attempt lines truncated for brevity) ... Attempt 52/120 - Task Status: Running Attempt 53/120 - Task Status: Succeeded SUCCESS: FDS configuration updated. deleteOrphanedImagesEnabled is now set to 'false'.
To disable automatic orphan cleanup:

./update-fds-oci-cleanup.sh false

Verification

After applying the configuration, you can verify the change by checking the Kubernetes ConfigMap:

Verify that fds.artifacts.oci.deleteOrphanedImages.enabled=true is rendered in the download-service-configs ConfigMap in the vcf-fleet-depot namespace:

kubectl get configmap download-service-configs -n vcf-fleet-depot -o yaml | grep deleteOrphanedImages

Expected Output:

fds.artifacts.oci.deleteOrphanedImages.enabled=true

Additional Information

Enabling automatic deletion reduces storage consumption in VCF Software Depot when orphaned images accumulate.
Disabling automatic deletion prevents loss of OCI images that were uploaded directly to the registry without invoking the VCF Software Depot upload API.