Supervisor sometimes stuck in configuring services with error "ReconcileFailed ... failed to verify certificate: x509: certificate signed by unknown"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

The supervisor cluster status shows intermittent "configuring" in vSphere UI under workload management.

 

 

GUI messages

Configured Core Supervisor Services

Service: tkg.vsphere.vmware.com. Reason: ReconcileFailed. Message: vendir: Error: Syncing directory '0': Syncing directory '.' with imgpkgBundle contents: Fetching image: Error while preparing a transport to talk with the registry: Unable to create round tripper: Get "https://docker-registry.kube-system.svc:5000/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority .

Service: velero.vsphere.vmware.com. Reason: ReconcileFailed. Message: vendir: Error: Syncing directory '0': Syncing directory '.' with imgpkgBundle contents: Fetching image: Error while preparing a transport to talk with the registry: Unable to create round tripper: Get "https://docker-registry.kube-system.svc:5000/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority .

 

Proxy is configured on Supervisor: Configure the Supervisor to Use a Proxy

 

ssh to the Supervisor shows:

 

kubectl get pkgi -n vmware-system-supervisor-services

NAME                            PACKAGE NAME                PACKAGE VERSION           DESCRIPTION                                                            AGE
svc-tkg.vsphere.vmware.com      tkg.vsphere.vmware.com      3.1.1-embedded            Reconcile failed: Error (see .status.usefulErrorMessage for details)   102d
svc-velero.vsphere.vmware.com   velero.vsphere.vmware.com   1.6.1-embedded+23741747   Reconcile failed: Error (see .status.usefulErrorMessage for details)   102d

kubectl describe pkgi svc-tkg.vsphere.vmware.com -n vmware-system-supervisor-services

Name:         svc-tkg.vsphere.vmware.com
Namespace:    vmware-system-supervisor-services
Labels:       appplatform.vmware.com/serviceId=tkg
              appplatform.vmware.com/serviceVersion=3.1.1-embedded
              managedBy=vSphere-AppPlatform
Annotations:  ext.packaging.carvel.dev/ytt-paths-from-secret-name: carvel-services-overlay
              packaging.carvel.dev/ignore-kubernetes-version-selection: true
API Version:  packaging.carvel.dev/v1alpha1
Kind:         PackageInstall
Metadata:
  Creation Timestamp:  2025-06-20T11:31:58Z
  Finalizers:
    finalizers.packageinstall.packaging.carvel.dev/delete
  Generation:        4
  Resource Version:  82880857
  UID:               ############
Spec:
  Package Ref:
    Ref Name:  tkg.vsphere.vmware.com
    Version Selection:
      Constraints:       3.1.1-embedded
  Service Account Name:  default-carvel-install-sa
  Values:
    Secret Ref:
      Name:  tkg.vsphere.vmware.com-3.1.1-embedded-config-secret-wof
    Secret Ref:
      Name:  tkg.vsphere.vmware.com-3.1.1-embedded-env-props-8lv
Status:
  Conditions:
    Message:               Error (see .status.usefulErrorMessage for details)
    Status:                True
    Type:                  ReconcileFailed
  Friendly Description:    Reconcile failed: Error (see .status.usefulErrorMessage for details)
  Last Attempted Version:  3.1.1-embedded
  Observed Generation:     4
  Useful Error Message:    vendir: Error: Syncing directory '0':
  Syncing directory '.' with imgpkgBundle contents:
    Fetching image:
      Error while preparing a transport to talk with the registry:
        Unable to create round tripper:
          Get "https://docker-registry.kube-system.svc:5000/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority

  Version:  3.1.1-embedded
Events:     <none>

 

 

Verify that the resourceVersion of the "kapp-controller-config" secret on the Supervisor cluster is changing every 5-20 seconds (it should be a constant value)

kubectl get secret -n vmware-system-appplatform-operator-system kapp-controller-config -o jsonpath="{.metadata.resourceVersion}" -w

 

######

Environment

vCenter 8.0U3

Cause

Mismatch in kapp-controller-config secret

Resolution

The problem will be solved in a future release

 

A workaround is restarting the responsible service on the Supervisor Control Plane nodes sequentially. This will make sure each service does read and have the latest correct values and prevent overwriting the previous settings. This restart must be applied every time the proxy settings or private container registry settings are modified on the Supervisor Cluster.

To apply the workaround, restart the following service as root on each of the three Supervisor Control Plane nodes:

systemctl restart wcp-sync

 

See also KB Inconsistent Proxy Settings and Trusted Certificates for Private Container Registries in Supervisor Cluster Causing imgpkg Failures and TKC Update Issues