"kubectl vsphere login" for both VKS Guest and Supervisor cluster fails with error "FATA[] Error while connecting to host <IP address>: internal server error"

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

• Trying to login to the VKS Supervisor or Guest Cluster via "kubectl vSphere login" fails with the error below.
  
  DEBU[YYYY-MM-DD <time>] Creating wcp.Client for <Supervisor IP>.          
There was an error when trying to connect to the server.
Please check the server URL and try again.
FATA[YYYY-MM-DD <time>] Error while connecting to host <Supervisor IP>: internal server error.


• The kubectl-plugin-vsphere pods confirm that the GET request for /wcp/loginbanner fails because of the SSL error due to an expired certificate.
  
  "GET /wcp/loginbanner HTTP/2.0" 502 157 "-" "kube-plugin-vsphere bld 23754142 - cln 13167650"
[error] 6#0: *293 SSL_read() failed (SSL: error:0A000415:SSL routines::sslv3 alert certificate expired:SSL alert number 45) while reading response header from upstream, client: <IP>, server: mgmt, request: "GET /wcp/loginbanner HTTP/2.0", upstream: "https://127.0.0.1:8443/wcp/loginbanner", host: "<supervisor-IP>"


• Performing a curl request manually towards https://<supervisor-IP>/wcp/loginbanner may or may not return an expired certificate.

Environment

VMware vSphere Kubernetes Service
VMware vCenter Server 7.x
VMware vCenter Server 8.x
VMware vCenter Server 9.x

Cause

The authproxy-client certificate inside the Supervisor Cluster is expired. To validate the same, run the following command in one of the Supervisor Control Plane Nodes.

find / -type f \( -name "*.cert" -o -name "*.crt" \)  -print 2>/dev/null | egrep -v 'ca.crt$|ca-bundle.crt$|kubelet\/pods|var\/lib\/containerd|run\/containerd' | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'

Resolution

Use the wcp_cert_manager tool to regenerate the expired authproxy-client certificate. The same can be downloaded from here- Replace vSphere with Tanzu Supervisor Certificates

Note: In case the authproxy-client certificate doesn't get regenerated using the tool, contact Broadcom Support to assist you with the Manual Method to re-generate this certificate. 

Additional Information

Master vSphere Supervisor Certificate Guide