dial tcp: lookup depot.kube-system.svc on 127.0.0.53:53: no such host

Products

VMware vSphere Kubernetes Service

Issue/Introduction

The following error is displayed while updating VKS Service on Supervisor.

Supervisor encountered an error while trying to reach the container image registry hosting Service Version 3.7.0+v1.36 and its signatures cannot be verified: Connectivity to image depot.kube-system.svc/vcf/supervisor-service-vks/ga/3.7.0/vsphere-kubernetes-service:3.7.0 cannot be verified: Get "https://depot.kube-system.svc/v2/": dial tcp: lookup depot.kube-system.svc on 127.0.0.53:53: no such host. Please check that the container image registry is configured properly and can be reached from the Supervisor's network.
This error occurs in environments where Regional Harbor is not configured.

Environment

vSphere Kubernetes Service

VCF 9.1

Cause

The issue occurs when the standard (non-legacy) VKS Service YAML definition is deployed on an environment lacking a configured Regional Harbor.

The standard VCF 9.1 YAML is strictly engineered for environments utilizing the newly integrated VCF Software Depot. This architecture relies on Regional Harbor to create and host the internal depot.kube-system.svc endpoint. Without an active Regional Harbor deployment, the internal DNS record does not exist. Consequently, the Supervisor fails to resolve the host when attempting to fetch the updated container images, resulting in registry authentication and name resolution failures.

Resolution

Utilizing the legacy YAML file initially prevents this failure by directing the cluster to external endpoints. However, if the standard YAML was already applied and the upgrade is stalled in a failed state, the local package must be manually patched.

The override-package-image.sh script must be used to correct the image repository URL for the imgpkg bundle and force reconciliation.

Note: These steps are strictly for updating the registry reference for the same service version and must not be used for rollbacks.

Verify the vSphere version hosting the Supervisor. For version 8.0P09 or later, the override-package-image.sh script is already present in the /usr/lib/vmware-wcp directory. For older versions, refer to Broadcom KB 442742 to obtain the script, and manually copy it into the /usr/lib/vmware-wcp directory on the Supervisor.
If the script was manually copied, grant execution permissions by running the following command:
```
chmod +x /usr/lib/vmware-wcp/override-package-image.sh
```
Retrieve the VKS package name by running the following command:
```
kubectl get package -n vmware-system-supervisor-services | grep tkg | grep <target version>
```
Note the package name from the output (e.g., tkg.vsphere.vmware.com.3.6.2+v1.35).

Perform a dry-run of the script to preview the configuration updates. Replace the package name and version numbers accordingly:

/usr/lib/vmware-wcp/override-package-image.sh -p <package name from the above command output> -i projects.packages.broadcom.com/vsphere/iaas/vsphere-kubernetes-service/3.x.x/vsphere-kubernetes-service:3.x.x --dry-run

Example:

{/usr/lib/vmware-wcp ]# /usr/lib/vmware-wcp/override-package-image.sh -p tkg.vsphere.vmware.com.3.6.2+v1.35 -i projects.packages.broadcom.com/vsphere/iaas/vsphere-kubernetes-service/3.6.2/vsphere-kubernetes-service:3.6.2 --dry-run

If the dry-run completes successfully without errors, execute the script to apply the changes and update the imgpkg bundle URL:

/usr/lib/vmware-wcp/override-package-image.sh -p <package name from the above command output> -i projects.packages.broadcom.com/vsphere/iaas/vsphere-kubernetes-service/3.x.x/vsphere-kubernetes-service:3.x.x

Example:

[/usr/lib/vmware-wcp ]# /usr/lib/vmware-wcp/override-package-image.sh -p tkg.vsphere.vmware.com.3.6.2+v1.35 -i projects.packages.broadcom.com/vsphere/iaas/vsphere-kubernetes-service/3.6.2/vsphere-kubernetes-service:3.6.2

Monitor the VKS service status to confirm it successfully reconciles on the correct version.