kubectl-plugin-vsphere failure - FATAL[] Error while connecting to host <SUPERVSIOR IP>: internal server error.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Authentication to vSphere Kubernetes Service (VKS) Supervisor or Guest Clusters fails - FATAL[] Error while connecting to host <SUPERVSIOR IP>: internal server error.

Analysis of kubectl-plugin-vsphere logs shows SSL handshake failures indicating an expired certificate on the login banner request:

2026/07/27 09:30:01 [error] 6#0: *765 SSL_do_handshake() failed (SSL: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired:SSL alert number 45) while SSL handshaking to upstream, client: <x.x.x.x>, server: default, request: "GET /wcp/loginbanner HTTP/1.1", upstream: "https://127.0.0.1:8443/wcp/loginbanner", host: "<x.x.x.x>"

Environment

vSphere Kubernetes Service (VKS) 8.x
Supervisor Cluster
VKS Guest Clusters

Cause

The authproxy-client.crt certificate on the local filesystem is expired or unsynchronized across the control plane nodes. A mismatch exists between the encoded certificate stored in the wcp-authproxy-client-secret Kubernetes secret and the certificate presented on the local filesystem at /etc/vmware/wcp/tls/authproxy-client.crt. Rotating only the spherelet certificates does not resolve this authentication proxy certificate mismatch.

Resolution

Perform the following steps on each Supervisor Control Plane VM to manually extract the synchronized certificates from the Kubernetes secret and replace the expired local files.

1. Verify Certificate Mismatch
   Execute the following commands on each control plane node to compare the expiration dates of the secret versus the local file. A mismatch confirms the desynchronization.
   
   
   • #Check secret expiration
     kubectl -n kube-system get secret wcp-authproxy-client-secret -o jsonpath='{.data.tls.crt}' | base64 -d | openssl x509 -noout -text | grep -B 1 "Not After"
   • # Check local file expiration
     openssl x509 -noout -text -in /etc/vmware/wcp/tls/authproxy-client.crt | grep -B 1 "Not After"

      2. Replace Authproxy-Client Certificates
          Extract the valid certificates and key from the wcp-authproxy-client-secret and overwrite the local TLS directory files:

• • kubectl -n kube-system get secret wcp-authproxy-client-secret -o jsonpath='{.data.tls.crt}' | base64 -d > /etc/vmware/wcp/tls/authproxy-client.crt
  • kubectl -n kube-system get secret wcp-authproxy-client-secret -o jsonpath='{.data.tls.key}' | base64 -d > /etc/vmware/wcp/tls/authproxy-client.key
  • kubectl -n kube-system get secret wcp-authproxy-client-secret -o jsonpath='{.data.ca.crt}' | base64 -d > /etc/vmware/wcp/tls/authproxy-client-ca.crt

       3. Restart Authentication Services
           Restart the kubectl-plugin-vsphere and wcp-authproxy services to load the newly written certificates.
           (Note: Use crictl or standard Kubernetes pod deletion to force a restart of these static/DaemonSet pods depending on the precise vSphere version footprint).

• • crictl rm -f $(crictl ps --label io.kubernetes.container.name=kubectl-plugin-vsphere -q)
  • crictl rm -f $(crictl ps --label io.kubernetes.container.name=wcp-authproxy -q)

        4. Validate Access
            Attempt authentication to the guest cluster using kubectl vsphere login to confirm the SSL handshake completes successfully and standard access is restored.

 

Additional Information

For standard Supervisor certificate replacement, reference KB 322994 (KB 322994 (Replace vSphere Supervisor (Previously known as vSphere with Tanzu) Certificates) ).

If ESXi hosts remain NotReady post-certificate replacement, verify spherelet logs (/var/log/spherelet.log) for lingering TLS constraints.