Supervisor Cluster stuck in configuring state with Configured Core Supervisor Services error

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• On the Workload Management menu in the vCenter Server user interface, the Supervisor Cluster remains stuck in a "Configuring" state and displays the following error: ERROR - Configured Core Supervisor Services
• Underlying package reconciliations fail with ReconcileFailed conditions for core supervisor services, including velero.vsphere.vmware.com and tkg.vsphere.vmware.com.
• Describing the package installs via kubectl reveals transport errors while attempting to fetch images from the registry:

#kubectl describe pkgi -n vmware-system-supervisor-services svc-velero.vsphere.vmware.com

status:
  conditions:
  - message: Error (see .status.usefulErrorMessage for details)
    status: "True"
    type: ReconcileFailed
  friendlyDescription: 'Reconcile failed: Error (see .status.usefulErrorMessage for details)'
  lastAttemptedVersion: 1.6.1-embedded+23741747
  observedGeneration: 1
  usefulErrorMessage: |
    vendir: Error: Syncing directory '0':
      Syncing directory '.' with imgpkgBundle contents:
        Fetching image:
          Error while preparing a transport to talk with the registry:
            Unable to create round tripper:
              Get "https://docker-registry.kube-system.svc:5000/v2/":
                Service Unavailable
  version: 1.6.1-embedded+23741747

 

#kubectl describe pkgi -n vmware-system-supervisor-services svc-tkg.vsphere.vmware.com

status:
  conditions:
  - message: Error (see .status.usefulErrorMessage for details)
    status: "True"
    type: ReconcileFailed
  friendlyDescription: 'Reconcile failed: Error (see .status.usefulErrorMessage for details)'
  lastAttemptedVersion: 3.6.2+v1.35
  observedGeneration: 10
  usefulErrorMessage: |
    kapp: Error: waiting on reconcile packageinstall/tanzu-cluster-api (packaging.carvel.dev/v1alpha1) namespace: svc-tkg-domain-c1018:
    Finished unsuccessfully (Reconcile failed: (message: vendir: Error: Syncing directory
      Syncing directory '.' with imgpkgBundle contents:
       Fetching image: Error while preparing a transport to talk with the registry:
        Unable to create round tripper:
          GET https://projects.packages.broadcom.com/artifactory/api/docker/projects/v2/token?scope=repository&service=projects.packages.broadcom.com::Authentication is required

Environment

vSphere Supervisor

Cause

Active HTTPS proxy settings on the vCenter Server block image fetching and package reconciliation within the underlying Workload Control Plane (wcp) service layer.

Resolution

1. Log into the vSphere Client and navigate to Supervisor > Configure > Network > Proxy configuration.

2. Temporarily disable the HTTPS proxy configuration.

3. Establish an SSH session to the vCenter Server Appliance as root and restart the Workload Control Plane service:

   service-control --restart wcp
   

4. Verify that the velero.vsphere.vmware.com and tkg.vsphere.vmware.com packages reconcile successfully.

5. Re-enable the HTTPS proxy configuration in the vSphere Client UI under Supervisor > Configure > Network > Proxy configuration.

6. Restart the wcp service again to apply the settings:

   service-control --restart wcp
   

Note: If package reconciliation does not occur after disabling proxy settings in the GUI and restarting wcp, completely remove the proxy settings from both the vSphere Client GUI and vCenter CLI before executing the wcp service restart.