OCI images uploaded manually to VCF Software Depot are not recognized by vcf-download-tool in air-gapped environments
search cancel

OCI images uploaded manually to VCF Software Depot are not recognized by vcf-download-tool in air-gapped environments

book

Article ID: 452070

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service VMware Cloud Foundation

Issue/Introduction

When deploying VKS in a VCF 9.1.0 air-gapped environment, OCI images for Supervisor Services and VKS Standard Packages fail to appear in the depot artifacts list output. This issue occurs after manually copying the images to the Software Depot's OCI registry using the oci_image_depot_migrator.py script (per steps 1c, 1d, 6a, and 6b of the air-gapped deployment guide).

Command:

vcf-download-tool depot artifacts list \
    --vcf-version=<vcf-version> --depot-fqdn=<software-depot-fqdn> \
    --ops-fqdn=<vcf-operations-fqdn> --ops-user=<ops-username> \
    --ops-user-password-file=<path-to-password-file>

Although the images are present in the Software Depot's OCI registry and function perfectly — allowing the Supervisor Services and VKS Standard Packages to install and run without issue — the vcf-download-tool does not track them.

For example, running the manage_depot_manual_oci_images.py check command against such a depot reports the image as unmanaged

$> ./manage_depot_manual_oci_images.py check

Software Depot: fleet.example.com
Scanned 3 image(s) across the OCI registry catalog.

Managed (2):
  [OK] vks-standard-packages/ga/3.6.0-20260211/vks-standard-packages:3.6.0-20260211  (component: VKS_STANDARD_PACKAGES)
  [OK] supervisor-service-contour/ga/1.33.1/contour:v1.33.1_vmware.1  (component: SUPERVISOR_SERVICE_CONTOUR)

Unmanaged (1) -- known component, not seen by vcf-download-tool:
  [!!] vcf-service-argocd/ga/1.1.0/argocd-service:v1.1.0_vmware.1  (component: SUPERVISOR_SERVICE_ARGOCD)

Action needed: see the unmanaged/unmapped sections above.

Environment

  • VMware Cloud Foundation (VCF) / VMware vSphere Foundation (VVF) 9.1.0, air-gapped deployment
  • VCF Software Depot (Fleet Depot Server)
  • vcf-download-tool (VCFDT)

This issue does not apply to:

  • VCF / VVF 9.1.1 and later, where the command, vcf-download-tool depot artifacts download and vcf-download-tool depot artifacts upload natively supports OCI image components (Supervisor Services, VKS Standard Packages); see the VKS Deployment Guide for VCF 9.1.1+ air-gapped environments.
  • Deployments using an external Enterprise OCI registry instead of the Software Depot (air-gapped.md, air-gapped-vcf90.md, air-gapped-harbor.md), which don't involve vcf-download-tool for OCI images at all.
  • Software Depots that have direct internet access and used vcf-download-tool (rather than oci_image_depot_migrator.py) to download and upload OCI images.

Cause

vcf-download-tool prior to VCF 9.1.1 does not support downloading and uploading OCI-image components (Supervisor Services, VKS Standard Packages) to the Software Depot. To make these images available in a disconnected or offline Software Depot on VCF 9.1.0, the VKS Deployment Guide for VCF 9.1.0 air-gapped environments instead uses the imgpkg-based oci_image_depot_migrator.py script to copy the images directly into the Software Depot's OCI registry.

vcf-download-tool's own artifacts download (to a local depot-store) and depot artifacts upload (from the depot-store to Software Depot) commands are the source of truth it uses to know which artifacts are present in the Software Depot.

  • Images pushed by oci_image_depot_migrator.py land in the same OCI registry but bypass the internal checks entirely, so they remain invisible to vcf-download-tool depot artifacts list even though they are physically present and fully usable by the Supervisor. An image in this state is referred to as unmanaged.
  • An image that vcf-download-tool depot artifacts list does report is managed.

Resolution

There are two options to resolve unmanaged images,

  • Making the image manageable (Option 1) is the recommended fix
  • Deleting the image (Option 2) is a workaround for cases where the image is no longer needed or you cannot run vcf-download-tool for that component.

Table of Contents


Prerequisites: Mandatory for both options

Important:

  • vcf-download-tool depot artifacts list and depot artifacts upload are the OCI/Carvel-artifact analog of the publicly documented depot binaries family (used for management-appliance ISOs) and take the same --vcf-version/--depot-fqdn/--ops-fqdn/--ops-user/--ops-user-password-file flags. The plain artifacts download command (no depot prefix — it downloads to a local directory, not to Software Depot) instead takes --depot-store and --depot-download-activation-code-file.
  • The first time vcf-download-tool talks to a given --depot-fqdn/--ops-fqdn pair, it interactively prompts to confirm the TLS certificate chain and to opt in/out of CEIP before it will proceed. manage_depot_manual_oci_images.py runs vcf-download-tool depot artifacts list with its stdin inherited from your terminal specifically so you can answer these prompts if they appear; run check interactively (not from a non-interactive script, cron job, or with stdin redirected from /dev/null) at least once per depot/ops-fqdn pair so you're present to answer them. Once accepted, vcf-download-tool does not prompt again for that same depot/ops-fqdn pair.
  • --vcf-version's default depends on the action:
    • For check, it defaults to 9.1 (the minor version, not a specific patch like 9.1.0 or 9.1.1), so vcf-download-tool depot artifacts list reports every image released under 9.1.x in one pass. Passing a specific patch version narrows what vcf-download-tool reports to that patch alone — an image that's genuinely managed but was released under a different 9.1.x patch than the one you passed would then be misreported as unmanaged.
    • For delete, it defaults to the specific patch 9.1.0 instead, since the untracked images this article covers were all manually uploaded under VCF 9.1.0 — a narrower, exact-patch check here keeps delete's internal re-scan scoped to that release rather than pulling in another 9.1.x patch's managed-version data.

Either default can be overridden with an explicit value if needed.

The remediation commands check prints for each unmanaged image use a separate flag, --remediation-vcf-version, defaulting to the specific patch 9.1.0 — unlike --vcf-version, this one must be an exact patch, not the 9.1 wildcard, since vcf-download-tool artifacts download/depot artifacts upload need a concrete release to fetch. 9.1.0 is correct by default because the manually-uploaded images this article covers were all built for that release; override --remediation-vcf-version only if you know a given image was actually released under a different 9.1.x patch.

Before applying either option, run check from a host with imgpkg-style network access to the Software Depot's registry endpoint and to VCF Operations to identify every unmanaged image:

## --vcf-version defaults to 9.1 (covers all 9.1.x patch releases); pass it only to narrow to one patch.
./manage_depot_manual_oci_images.py check \
    --depot-fqdn <software-depot-fqdn> \
    --ops-fqdn <vcf-operations-fqdn> --ops-user <ops-username> \
    --ops-user-password-file <path-to-password-file>

## Sample Command
./manage_depot_manual_oci_images.py check \
    --depot-fqdn fleet.example.com \
    --ops-fqdn ops.example.com --ops-user [email protected] \
    --ops-user-password-file /root/.ops-pw

 

Sample output (first run against a depot/ops-fqdn pair, showing the TLS-certificate prompts)

manage_depot_manual_oci_images.py streams vcf-download-tool's output live and passes your terminal's stdin straight through to it, so when the prompts described above appear, answer them exactly as you would running vcf-download-tool directly (the Y after each prompt below is the answer you type, not something the tool printed).
This run passes --vcf-version 9.1.1 explicitly to narrow the check to that one patch; omit it (as in the Sample Command above) to check against every 9.1.x release at once using the default.

$ ./manage_depot_manual_oci_images.py check \
    --depot-fqdn fleet.example.com --vcf-version 9.1.1 \
    --ops-fqdn ops.example.com --ops-user admin --ops-user-password-file /home/worker/tmp/.ops-pw
Warning: TLS certificate verification is disabled for Software Depot registry calls (matches the 'curl -k' convention used by the other airgapped/scripts/*.sh scripts).
+ vcf-download-tool depot artifacts list --vcf-version=9.1.1 --depot-fqdn=fleet.example.com --ops-fqdn=ops.example.com --ops-user=admin --ops-user-password-file=/home/worker/tmp/.ops-pw
*********Welcome to VCF Download Tool***********

Chain of certificates from ##.###.##.###:
   1: OU=Broadcom\, Inc.,O=Broadcom\, Inc.,CN=VCFOps-slice-1
   2: OU=Broadcom\, Inc.,O=Broadcom\, Inc.,CN=VCFOps-cluster-ca_########-####-####-####-############
Confirm certificates or choose certificate number to review [1-2/Y/N]:Y
Certificate:
    ... (certificate details, elided for brevity) ...
Confirm certificate from fleet.example.com [Y/N]Y
Version: 9.1.1.0.25665410
Getting artifacts list from the depot.

Waiting for query ########-####-####-####-############ to complete (elapsed: 0 seconds)...
Waiting for query ########-####-####-####-############ to complete (elapsed: 4 seconds)...
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ID                                   | Component                 | Component Full Name          | Version               | Size*      | Release Date | OCI Image Count | Is Partial
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
########-####-####-####-############ | SUPERVISOR_SERVICE_HARBOR | Harbor Service               | 2.15.2+vmware.1-vks.1 |  119.2 KiB | DD/MM/YYYY   | 1/1             | false
########-####-####-####-############ | SUPERVISOR                | VMware vSphere Supervisor    | 9.1.1.0.25667503      |    3.8 GiB | DD/MM/YYYY   |                 | false
########-####-####-####-############ | DSM                       | VMware Data Services Manager | 9.1.1.0.25662079      |   32.9 GiB | DD/MM/YYYY   | 1/1             | false
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
3 elements
* Note: Size does not include the sizes of component OCI images.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Log file: /home/worker/fds/log/vdt.log
Software Depot: fleet.example.com
Scanned 53 image(s) across the OCI registry catalog.

Managed (24):
  [OK] supervisor-service-harbor/ga/2.15.2/harbor:v2.15.2_vmware.1-vks.1  (component: SUPERVISOR_SERVICE_HARBOR)  (+16 other tag(s) in this image repo)
  [OK] vcf-service-data-services/ga/9.1.1.0/dsm-consumption-operator-supervisor:9.1.1.0.25623779  (component: DSM)  (+6 other tag(s) in this image repo)

Unmanaged (29) -- known component, not seen by vcf-download-tool for --vcf-version=9.1.1 (may be available under a different VCF version):
  [!!] supervisor-service-harbor/ga/2.14.2/harbor:v2.14.2_vmware.2-vks.1  (component: SUPERVISOR_SERVICE_HARBOR)  (+16 other tag(s) in this image repo)
  [!!] vcf-service-argocd/ga/1.1.0/argocd-service:v1.1.0_vmware.1  (component: SUPERVISOR_SERVICE_ARGOCD)  (+11 other tag(s) in this image repo)

Remediation commands (run on a host with vcf-download-tool and network access to VCF Operations):

... (one two-step download/upload command pair per unmanaged repo -- see the format under "Option 1" below) ...

Action needed: see the unmanaged section above.

If you're redirecting check's output to a file or tee for a record, expect the certificate dump and these prompts to appear inline with the rest of the output, since stdout and stderr are merged and streamed live for exactly this reason -- so that the prompts are visible in time for you to answer them, instead of being buffered until the command exits.

check matches each repo path discovered in the Software Depot's OCI registry catalog to a vcf-download-tool --component value using the table below (sourced from the external_repo field of each component in Broadcom's UPT packages-publish manifest for the release this table was last refreshed against).

--componentSoftware Depot repo-path prefix
SUPERVISOR_SERVICE_ARGOCD/vcf-service-argocd/ga
SUPERVISOR_SERVICE_HARBOR/supervisor-service-harbor/ga
SUPERVISOR_SERVICE_LCI/supervisor-service-lci/ga
VCF_CONSUMPTION_CLI_PLUGINS/vcf-cli-plugins/ga
VCF_SERVICE_CONFIGURATION/vcf-service-configuration/ga
SUPERVISOR_SERVICE_CONTOUR/supervisor-service-contour/ga
VCF_SERVICE_SECRET_STORE/vcf-service-secret-store/ga
VKS_STANDARD_PACKAGES/vks-standard-packages/ga
SUPERVISOR_SERVICE_METRICS_AGGREGATOR/supervisor-service-metrics-aggregator/ga
SUPERVISOR_SERVICE_EXTDNS/supervisor-service-extdns/ga
SUPERVISOR_SERVICE_VKS/supervisor-service-vks/ga
SUPERVISOR_SERVICE_SUPERVISOR_MANAGEMENT_PROXY/supervisor-service-supervisor-management-proxy/ga
SUPERVISOR_SERVICE_CA_CLUSTERISSUER/supervisor-service-ca-clusterissuer/ga
VKSM_EXTENSIONS/vksm-extensions/ga
VCF_SERVICE_PROTECTION_AND_RECOVERY/vcf-service-protection-and-recovery/ga
VKR/vsphere-kubernetes-release/ga
DSM/vcf-service-data-services/ga
VCF_SERVICE_HARBOR/vcf-service-harbor/ga
VCF_SERVICE_METRICS_AGGREGATOR/vcf-service-metrics-aggregator/ga
VCF_SERVICE_MIGRATION/vcf-service-migration/ga
VCF_SERVICE_ENCRYPTION_MANAGEMENT/vcf-service-encryption-management/ga
VCF_SERVICE_VKSM_AUTO_ATTACH/vcf-service-vksm-auto-attach/ga

Note: 

VCF_SERVICE_DATA_SERVICES is intentionally not in this table: its repo path (/vcf-service-data-services/ga) is identical to DSM's, and this script's prefix-only matching can't tell the two components' images apart by filename further down the path.
A VCF_SERVICE_DATA_SERVICES image would surface as unmapped rather than being misattributed to DSM.

A repo path that matches none of these prefixes is reported as unmapped. This is an informational warning, not an error — it does not affect check's exit code, and check does not suggest remediation or deletion for unmapped images. This table is a static snapshot and will inevitably lag whatever Broadcom actually ships: an unmapped image could be a component released after this table was last refreshed, just as easily as it could be a foreign/unrelated image.

For example:

Unmapped (1) -- not matched to a known component in this script's COMPONENT_REPO_PREFIXES table (informational warning, not an error):
  [??] some-new-service/ga/1.0.0/some-new-service:1.0.0

This does not necessarily indicate a problem -- COMPONENT_REPO_PREFIXES can lag behind the
actual set of released components, so these could be valid images from a component added
after this table was last updated. No remediation or deletion is suggested for unmapped
images; investigate only if a repo path looks unfamiliar or unexpected.

check exits 0 when everything is managed and 1 when action is needed, so it can be used as a gate in a script or pipeline.

Option 1 (Recommended): Make the image manageable by vcf-download-tool

For each unmanaged image, check automatically prints a ready-to-run, two-step remediation command pair:

Remediation commands (run on a host with vcf-download-tool and network access to VCF Operations):

# Unmanaged image: fleet.example.com/vcf-service-argocd/ga/1.1.0/argocd-service:v1.1.0_vmware.1
# (matched component: SUPERVISOR_SERVICE_ARGOCD; depot tag found: v1.1.0_vmware.1)
# NOTE: vcf-download-tool's depot-artifacts commands take --vcf-version as the
# VCF release identifier, not a per-image version; the tag above is shown so
# you can visually confirm it matches what --vcf-version=9.1.0 will fetch.
vcf-download-tool depot artifacts download --component=SUPERVISOR_SERVICE_ARGOCD --vcf-version=9.1.0 \
    --ops-fqdn=ops.env1.lab.test [email protected] --ops-user-password-file=/root/.ops-pw
vcf-download-tool depot artifacts upload --component=SUPERVISOR_SERVICE_ARGOCD --vcf-version=9.1.0 \
    --depot-fqdn=fleet.example.com \
    --ops-fqdn=ops.example.com [email protected] --ops-user-password-file=/root/.ops-pw
 

--vcf-version in these two commands is always the VCF release identifier, taken from --remediation-vcf-version (9.1.0 by default), not a per-image version — vcf-download-tool has no separate per-artifact version flag for this command family. The depot tag check found for the unmanaged image is printed as an informational comment above the commands so you can confirm it's the version you expect before running download/upload

After running both commands, re-run check. When every image is managed, it prints:

✅ All 3 image(s) in Software Depot are managed by vcf-download-tool.
 

Option 2 (Workaround): Delete the unmanaged image

Important

Deleting an image manifest is destructive, production-impacting, and effectively irreversible. Any Supervisor or VKS deployment that still pulls this image by tag will fail after deletion. Use this only if you cannot or do not want to run vcf-download-tool for the affected component — Option 1 is always the preferred path.

Warning: Deleting a manifest only unlinks it from the registry's tag list; the underlying image blobs — and the repo path's own entry in the registry's _catalog listing — are reclaimed only by a separate registry garbage-collection pass, which this script does not perform. It's expected for a repo to still appear in _catalog with {"tags":null} after a full deletion; check already treats a zero-tag repo as gone (it won't show up as managed/unmanaged/unmapped), so this is harmless and requires no action unless you want Software Depot's own storage/catalog fully reclaimed (a registry-administration task, outside what this script's delete can do).

Deletion is gated behind the same toggle_software_depot_oci_image_upload.sh script used in air-gapped-vcf91.md steps 5c/6c: delete calls it with enable before deleting anything, and with disable afterward no matter what (success, failure, or interruption), so the depot's OCI registry is never left open longer than necessary. This requires the VSP host and admin credentials already used with that script.

delete requires exactly one of --all (every unmanaged image) or --repos (a comma-separated list of specific repo paths, for when you don't want to delete every unmanaged image) to select scope — there is no implicit "delete everything" default. --repos takes the bare repo path — the exact value check prints on its "To target only this repo with 'delete': --repos ..." hint line — not the / form shown in the "Unmanaged images under:" line above it (the tool tolerates that prefix if pasted by mistake, but the canonical form is the bare path).

## --vcf-version defaults to 9.1.0 for 'delete' (see the note below); pass it only to override.
./manage_depot_manual_oci_images.py delete \
    --depot-fqdn <software-depot-fqdn> \
    --ops-fqdn <vcf-operations-fqdn> --ops-user <ops-username> \
    --ops-user-password-file <path-to-password-file> \
    --vsp-host <vsp-host-fqdn> --admin-username <admin-username> --admin-password '<admin-password>' \
    (--all | --repos <repo-path>[,<repo-path>...]) [--tag <tag>] [--dry-run]

## Sample Command: delete a specific repo (dry run first, strongly recommended)
./manage_depot_manual_oci_images.py delete \
    --depot-fqdn fleet.example.com \
    --ops-fqdn ops.example.com--ops-user [email protected] \
    --ops-user-password-file /root/.ops-pw \
    --vsp-host vsp.example.com --admin-username [email protected] --admin-password '<MASKED PASSWORD>' \
    --repos vcf-service-argocd/ga/1.1.0/argocd-service --dry-run

## Sample output
1 unmanaged image(s) selected for deletion:
  - vcf-service-argocd/ga/1.1.0/argocd-service  (component: SUPERVISOR_SERVICE_ARGOCD)  (tag: v1.1.0_vmware.1)

--dry-run: no confirmation prompt, no toggle call, and no DELETE requests were made.
 

--repos accepts a comma-separated list to target several specific repos in one run (e.g. --repos vcf-service-argocd/ga/1.1.0/argocd-service,supervisor-service-harbor/ga/2.14.2/harbor). To delete every unmanaged image found by check instead, use --all in place of --repos.

Once you've confirmed the plan with --dry-run, re-run without it. You must type the literal word DELETE to proceed:

*** WARNING: DESTRUCTIVE, PRODUCTION-IMPACTING, IRREVERSIBLE OPERATION ***
This will permanently delete the following image manifest(s) from the
Software Depot OCI registry. ...

  - vcf-service-argocd/ga/1.1.0/argocd-service : v1.1.0_vmware.1

Type DELETE (all caps) to proceed, anything else aborts: DELETE
+ toggle_software_depot_oci_image_upload.sh enable --vsp-host ... --admin-username ... --admin-password ****
...
Deleted vcf-service-argocd/ga/1.1.0/argocd-service:v1.1.0_vmware.1 (digest sha256:...).
+ toggle_software_depot_oci_image_upload.sh disable --vsp-host ... --admin-username ... --admin-password ****
...

Summary: 1 succeeded, 0 failed.

Anything other than the exact word DELETE (including pressing Enter with no input) aborts before the registry is ever toggled open. For scripted use, --yes-i-am-sure DELETE skips the interactive prompt but still requires that exact value.

When a repo has multiple tags (e.g. an image plus its cosign .sig/.imgpkg/.image-locations.imgpkg companion tags), some of them commonly share the same underlying manifest digest. Deleting one such tag can make another tag in the same batch stop resolving before the script gets to it — this shows up as Already gone (no longer resolves, nothing to delete): :. and is counted toward the succeeded total (e.g. Summary: 8 succeeded (3 of which were already gone), 0 failed.), not as a failure, since the desired end state — the tag is gone — is already met.

After the deletion completes, run ./manage_depot_manual_oci_images.py check again to see which unmanaged images (if any) still remain, and iterate — remediating each one via Option 1 or deleting it via this option — until check reports everything as managed or intentionally unmapped.

After remediation: enable automatic orphan cleanup

Whichever option you used above, after all manually uploaded OCI images are migrated or cleaned properly, the Software Depot's OCI registry can still accumulate genuinely orphaned OCI images over time — for reasons unrelated to the manual-upload scenario this article covers (for example, blobs or manifests left behind by interrupted or superseded uploads). Detecting and removing those is outside the scope of this article and of manage_depot_manual_oci_images.py.

Once you've finished remediating or deleting the manually-uploaded images above, enable the VCF software depot's built-in automatic orphan cleanup by following KB, Managing Automatic Cleanup of Orphaned OCI Images in VCF Software Depot, so that real orphaned OCI images on the Software Depot are cleaned up properly going forward.

Additional Information

Powered by Wolken Software