VKS and other Supervisor Service fails to reconcile when trying to reach an incorrect image depot "depot.kube-system.svc"

Products

VMware vSphere Kubernetes Service

Issue/Introduction

vSphere Kubernetes Service (VKS) upgrade from version fails to complete. In the vSphere UI Workload Management view, the upgrade remains stuck at 17 of 18 conditions completed.

Please Note that you may encounter similar errors while deploying any other supervisor services

One of the following errors observed on the Supervisor under Configured Core Supervisor Services:

Service: tkg.vsphere.vmware.com
Reason: ReconcileFailed
Message: vendir: Error: Syncing directory '0': 
Syncing directory '.' with imgpkgBundle contents: 
Fetching image: Error while preparing a transport 
to talk with the registry: Unable to create round tripper: 
Get "https://depot.kube-system.svc/v2/": Forbidden

Reason: ReconcileFailed. Message: vendir: Error: Syncing directory '0': Syncing directory '.' with imgpkgBundle contents: Fetching image: Error while preparing a transport to talk with the registry: Unable to create round tripper: Get "https://depot.kube-system.svc/v2/": dial tcp: lookup depot.kube-system.svc on 127.0.0.53:53: no such host

Environment

vSphere Kubernetes Service (VKS)

vSphere Supervisor Services

Cause

This issue occurs when the standard (non-legacy) VKS Service YAML definition is applied to a pre-VCF 9.1.0 or air-gapped environment.

The standard YAML is designed for environments supporting the VCF Software Depot and attempts to fetch the required image from an internal repository (depot.kube-system.svc). Because this internal depot is not available or resolvable in older architectures, it results in registry authentication and name resolution failures.

For pre-VCF 9.1.0 deployments, the 'legacy' version of the YAML must be applied, as it correctly directs the service to fetch the image from the external Broadcom repository (projects.packages.broadcom.com).

To avoid registry authentication failures, ensure the correct YAML definition is applied based on your VCF version:

Legacy yaml (e.g., vsphere-kubernetes-service-legacy-3.6.2+v1.35.yaml) : Points to the public 'projects.packages.broadcom.com' instance via its imgpkgBundle. This must only be used for vCenter versions 9.0 and lower, or within air-gapped vCenter 9.1 (or newer) environments.
Standard yaml (e.g., vsphere-kubernetes-service-3.6.2+v1.35.yaml): Points to the VCF Operations Software Depot via its imgpkgBundle. This must be used for internet-connected vCenter versions 9.1 and newer.

Resolution

To resolve this issue, use the override-package-image.sh script to correct the image repository URL for the imgpkg bundle and force reconciliation.

PLEASE NOTE: These steps are strictly for updating the registry reference for the same service version and should not be used for any rollbacks.

Verify the vSphere version where your Supervisor is running. If you are running version 8.0P09 or later, the override-package-image.sh script is already present under the /usr/lib/vmware-wcp directory. For older versions, manually copy the provided script onto the Supervisor into the /usr/lib/vmware-wcp directory.
If you manually copied the script, grant execution permissions by running the following command:
```
chmod +x /usr/lib/vmware-wcp/override-package-image.sh
```
Retrieve the VKS package name by running the following command:
```
kubectl get package -n vmware-system-supervisor-services | grep tkg | grep <target version>
```
Note the package name from the output (e.g., tkg.vsphere.vmware.com.3.6.2+v1.35).

Perform a dry-run of the script to preview the configuration updates. Replace the package name in the command below if yours differs:

/usr/lib/vmware-wcp/override-package-image.sh -p <package name from the above command output> -i projects.packages.broadcom.com/vsphere/iaas/vsphere-kubernetes-service/3.x.x/vsphere-kubernetes-service:3.x.x --dry-run

Example:

{/usr/lib/vmware-wcp ]# /usr/lib/vmware-wcp/override-package-image.sh -p tkg.vsphere.vmware.com.3.6.2+v1.35 -i projects.packages.broadcom.com/vsphere/iaas/vsphere-kubernetes-service/3.6.2/vsphere-kubernetes-service:3.6.2 --dry-run

If the dry-run completes successfully without errors, execute the script to apply the changes and update the imgpkg bundle URL:

/usr/lib/vmware-wcp/override-package-image.sh -p <package name from the above command output> -i projects.packages.broadcom.com/vsphere/iaas/vsphere-kubernetes-service/3.x.x/vsphere-kubernetes-service:3.x.x

Example:

[/usr/lib/vmware-wcp ]# /usr/lib/vmware-wcp/override-package-image.sh -p tkg.vsphere.vmware.com.3.6.2+v1.35 -i projects.packages.broadcom.com/vsphere/iaas/vsphere-kubernetes-service/3.6.2/vsphere-kubernetes-service:3.6.2

Monitor the VKS service status to confirm it successfully reconciles on the correct version.

Additional Information

Refer VMware vSphere Kubernetes Service Release Notes
Resolving other depot reachability errors: The override-package-image.sh script can also be used to resolve ReconcileFailed errors for other Supervisor Services if the error indicates an inability to reach the image depot. When doing so, simply adjust the grep filter in the command from Step 3 of the Resolution to search for the specific package name you are troubleshooting.
Script usage: To view the full usage instructions and available options for the script, run the following command with the -h flag:
```
/usr/lib/vmware-wcp/override-package-image.sh -h
```

Attachments

override-package-image.sh get_app