VKS and other Supervisor Services fail to reconcile when trying to reach an incorrect image depot "depot.kube-system.svc"
search cancel

VKS and other Supervisor Services fail to reconcile when trying to reach an incorrect image depot "depot.kube-system.svc"

book

Article ID: 442742

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

  • The vSphere Kubernetes Service (VKS) upgrade fails to complete. In the vSphere UI Workload Management view, the upgrade remains stuck at 17 of 18 conditions completed.
  • Note that you may encounter similar errors while deploying any other supervisor services.
  • One of the following errors is observed on the Supervisor under Configured Core Supervisor Services:
    Service: tkg.vsphere.vmware.com
    Reason: ReconcileFailed
    Message: vendir: Error: Syncing directory '0': Syncing directory '.' with imgpkgBundle contents: Fetching image: Error while preparing a transport to talk with the registry: Unable to create round tripper: Get "https://depot.kube-system.svc/v2/": Forbidden

    Reason: ReconcileFailed. Message: vendir: Error: Syncing directory '0': Syncing directory '.' with imgpkgBundle contents: Fetching image: Error while preparing a transport to talk with the registry: Unable to create round tripper: Get "https://depot.kube-system.svc/v2/": dial tcp: lookup depot.kube-system.svc on 127.0.0.53:53: no such host

Environment

  • vSphere Kubernetes Service (VKS)
  • vSphere Supervisor Services

Cause

This issue occurs for two different reasons:

  • When the standard (non-legacy) VKS Service YAML definition is applied to a VCF 9.0 or air-gapped environment.
    • The standard YAML is designed for environments supporting the VCF Software Depot and attempts to fetch the required image from an internal repository (depot.kube-system.svc). Because this internal depot is not available or resolvable in older architectures, it results in registry authentication and name resolution failures.
    • For pre-VCF 9.1.0 deployments, the 'legacy' version of the YAML must be applied, as it correctly directs the service to fetch the image from the external Broadcom repository (projects.packages.broadcom.com).
    • To avoid registry authentication failures, ensure the correct YAML definition is applied based on your VCF version:
      • Legacy yaml (e.g., vsphere-kubernetes-service-legacy-3.6.2+v1.35.yaml): Points to the public 'projects.packages.broadcom.com' instance via its imgpkgBundle. This must only be used for vCenter versions 9.0 and lower, or within air-gapped vCenter 9.1 (or newer) environments.
      • Standard yaml (e.g., vsphere-kubernetes-service-3.6.2+v1.35.yaml): Points to the VCF Operations Software Depot via its imgpkgBundle. This must be used for internet-connected vCenter versions 9.1 and newer.

 

  • When the regional Harbor is missing for VCF 9.1.0 environments.
    • The VCF Harbor or the Regional Harbor is required in VCF 9.1 for using services dependent on depot.kube-system.svc.
    • The VCF Harbor or the Regional Harbor serves the depot.kube-system.svc endpoint. The Regional Harbor is deployed through VCFA and contains this without any user customization.
    • A manual Harbor install is required for VVF.

Resolution

Resolution for when the standard (non-legacy) VKS Service YAML definition is applied to a pre-VCF 9.1.0 or air-gapped environment

DO NOT USE THIS OPTION FOR VCF/VVF 9.1.0 NON-AIR-GAPPED ENVIRONMENTS

To resolve this issue, use the override-package-image.sh script to correct the image repository URL for the imgpkg bundle and force reconciliation.

Note: These steps are strictly for updating the registry reference for the same service version and should not be used for any rollbacks.

  1. Run the following commands directly from the Supervisor control plane.
    Note: Check KB SSH into the Supervisor control plane for instructions to log in to the Supervisor Control Plane.
  2. Verify the vCenter server version where the Supervisor is configured. If the VCenter server version is 8.0U3j or later, the override-package-image.sh script is already present under the /usr/lib/vmware-wcp directory. For older versions, manually copy the provided script in this KB onto the Supervisor into the /usr/lib/vmware-wcp directory.

  3. If you manually copy the script, grant execution permissions by running the following command:
    chmod +x /usr/lib/vmware-wcp/override-package-image.sh

  4. Retrieve the VKS package name by running the following command:
    kubectl get package -n vmware-system-supervisor-services | grep tkg | grep <target version>

    Note the package name from the output (e.g., tkg.vsphere.vmware.com.3.6.2+v1.35).

  5. Perform a dry-run of the script to preview the configuration updates. Replace the package name in the command below if yours differs:
    /usr/lib/vmware-wcp/override-package-image.sh -p <package name from the above command output> -i projects.packages.broadcom.com/vsphere/iaas/vsphere-kubernetes-service/3.#.#/vsphere-kubernetes-service:3.#.# --dry-run

    Example:
    [/usr/lib/vmware-wcp]# /usr/lib/vmware-wcp/override-package-image.sh -p tkg.vsphere.vmware.com.3.6.2+v1.35 -i projects.packages.broadcom.com/vsphere/iaas/vsphere-kubernetes-service/3.6.2/vsphere-kubernetes-service:3.6.2 --dry-run

  6. If the dry-run completes successfully without errors, execute the script to apply the changes and update the imgpkg bundle URL:
    /usr/lib/vmware-wcp/override-package-image.sh -p <package name from the above command output> -i projects.packages.broadcom.com/vsphere/iaas/vsphere-kubernetes-service/3.#.#/vsphere-kubernetes-service:3.#.#

    Example:
    [/usr/lib/vmware-wcp]# /usr/lib/vmware-wcp/override-package-image.sh -p tkg.vsphere.vmware.com.3.6.2+v1.35 -i projects.packages.broadcom.com/vsphere/iaas/vsphere-kubernetes-service/3.6.2/vsphere-kubernetes-service:3.6.2

  7. Monitor the VKS service status to confirm it successfully reconciles on the correct version.

 

Resolution when the regional Harbor is missing for VCF 9.1.0 environments that are not air-gapped.

  1. VCFA regional harbor deployment:
    Install Regional Harbor Through VCF Automation
  2. VVF harbor deployment:
    Deploy Harbor Supervisor Service in vSphere Foundation

Additional Information

  • Refer to the VMware vSphere Kubernetes Service Release Notes
  • Resolving other depot reachability errors: The override-package-image.sh script can also be used to resolve ReconcileFailed errors for other Supervisor Services if the error indicates an inability to reach the image depot. When doing so, adjust the grep filter in the command from Step 3 of the Resolution to search for the specific package name you are troubleshooting.

  • Script usage: To view the full usage instructions and available options for the script, run the following command with the -h flag:
    /usr/lib/vmware-wcp/override-package-image.sh -h

  • If the following error message returns when attempting to run the script, it indicates the script contains carriage returns (CRLF) that has to be removed in order for the script to run successfully.
    /bin/bash^M: bad interpreter: No such file or directory

  • In order to resolve this error, run the following sed command to remove the CRLF carriage returns.
    sed -i 's/\r$//' /usr/lib/vmware-wcp/override-package-image.sh

Attachments

override-package-image.sh get_app
Powered by Wolken Software