Deployment of Supervisor Services Fails with TLS Certificate Validation Error: "x509: certificate is not valid for any names, but wanted to match mgmt-image-proxy.kube-system.svc.cluster.local"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• When attempting to deploy Supervisor services (e.g., Harbor, Configuration Service, VKS Cluster Management), deployment fails or remains stuck in an error state.
• The Supervisor displays certificate validation warnings or signature validation errors:

Supervisor encountered an error while trying to reach the container image registry hosting Service Version <version> and its signatures cannot be verified: Connectivity to image mgmt-image-proxy.kube-system.svc.cluster.local/configuration.vsphere.vmware.com-<version>@sha256:<hash> cannot be verified: Get "https://mgmt-image-proxy.kube-system.svc.cluster.local/v2/": tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match mgmt-image-proxy.kube-system.svc.cluster.local; Get "https://mgmt-image-proxy.kube-system.svc.cluster.local/v2/": tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match mgmt-image-proxy.kube-system.svc.cluster.local Please check that the container image registry is configured properly and can be reached from the Supervisor's network.
Running unverified services on user workloads has security risks. An unverified service has network access to user workloads, vSphere Pods, and exposed APIs.
Signature Certificate is not available.

• Proceeding despite warnings leaves the service status as ReconcileFailed with errors like:

Reason: ReconcileFailed. Message: vendir: Error: Syncing directory '0': Syncing directory "' with imgpkgBundle contents: Fetching image: Error while preparing a transport to talk with the registry: Unable to create round tripper: GET https://mgmt-image-proxy.kube-system.svc.cluster.local/oauth/provider/token?scope=repository%3Aconfiguration.vsphere.vmware.com-<version>%3Apull&service =: unexpected status code 405 Method Not Allowed: {"error":"client_error","error_description":"HTTP 405 Method Not Allowed","error_uri":null,"minorErrorCode":"client_error","message":"HTTP 405 Method NotAllowed","stackTrace":null}.

• Inspecting ESXi host logs (/var/run/log/crx/imagefetcher/guest.log) shows similar TLS validation failures while trying to resolve mgmt-image-proxy:

<TIMESTAMP> vmx <PID> [esx@<ID>] time="<TIMESTAMP>" level=error msg="Failed to resolve image mgmt-image-proxy.kube-system.svc.cluster.local/harbor.tanzu.vmware.com-<version>@sha256:<hash>. Err failed to do request: Head \"https://mgmt-image-proxy.kube-system.svc.cluster.local/v2/harbor.tanzu.vmware.com-<version>/manifests/sha256:<hash>\": tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match mgmt-image-proxy.kube-system.svc.cluster.local"

Environment

VCF 9.X

Cause

• Manual or incorrect DNAT / SNAT rules configured in NSX Network Services cause traffic directed to the image proxy's external IP to be redirected to the kube-apiserver VIP (or another unintended endpoint) instead.
• When an ESXi host or component attempts to complete a TLS handshake with the proxy's IP, it receives the SSL/TLS certificate of kube-apiserver instead of mgmt-image-proxy, resulting in the x509: certificate is not valid hostname mismatch.

Resolution

• Identify the Cluster IP and External IP for mgmt-image-proxy: Run the following command on the Supervisor Control Plane:

kubectl -n kube-system get svc mgmt-image-proxy -o wide

• Test Certificate Resolution from the ESXi Host: Run openssl directly from an affected ESXi host against the proxy's external IP:

openssl s_client -connect <EXTERNAL-IP>:443 -servername mgmt-image-proxy.kube-system.svc.cluster.local | openssl x509 -noout -text | grep -A 2 "Subject Alternative Name"

depth=1 CN = kubernetes
verify return:1
depth=0 C = US, ST = CA, L = Palo Alto, O = VMware, OU = VMware Engineering, CN = mgmt-image-proxy-<ID>
verify return:1
            X509v3 Subject Alternative Name:
                IP Address:127.0.0.1, DNS:mgmt-image-proxy.kube-system.svc.cluster.local
    Signature Algorithm: sha256WithRSAEncryption
    

• Compare with Direct Supervisor Resolution:Running the same command directly on the Supervisor Control Plane should properly return the proxy's CN and SAN:

openssl s_client -connect <EXTERNAL-IP>:443 -servername mgmt-image-proxy.kube-system.svc.cluster.local | openssl x509 -noout -text | grep -A 2 "Subject Alternative Name"

depth=1 CN = CA, DC = vsphere, DC = local, C = US, ST = California, O = <ORG_NAME>.<DOMAIN>, OU = VMware Engineering
verify return:1
depth=0 OU = <ORG_NAME>.<DOMAIN>, CN = kube-apiserver-<ID>
verify return:1
            X509v3 Subject Alternative Name:
                IP Address:<SUPERVISOR_VIP>
            X509v3 Key Usage: critical

• Check NSX NAT Rules:

1. 1. Navigate to NSX UI > Networking > Network Services > NAT.

   2. Look for manual DNAT or SNAT rules overriding traffic destined for the Supervisor Cluster VIPs.

Resolution

1. Log into the NSX Manager UI.

2. Go to Networking > NAT Rules and select the affected VPC/Router context.

3. Locate and select any manual or conflicting SNAT / DNAT rules affecting the Supervisor VIPs or image proxy IP addresses.

4. Disable or Delete the conflicting manual NAT rules.

5. Re-verify connectivity from the ESXi host using the openssl command in step 2 to ensure the certificate now resolves to mgmt-image-proxy.

6. Retry the Supervisor Service deployment.