vSphere with Tanzu TKC login fails with "Login failed: bad gateway"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• When try to login TKC in vSphere with Tanzu, fails with below messages:

# kubectl vsphere login --server <KUBERNETES-CONTROL-PLANE-IP-ADDRES> --tanzu-kubernetes-cluster-namespace <KUBERNETES-NAMESAPCE> --tanzu-kubernetes-cluster-name <TKC-NAME> -u <TKC-USER-NAME> 
Error Message:
KUBECTL_VSPHERE_PASSWORD environment variable is not set. Please enter the password below 
Password: 
ERRO[0006] Login failed: bad gateway 
ERRO[0006] Failed login to Tanzu Kubernetes cluster <TKC-NAME>: bad gateway 

• Check the container status on each Supervisor VM with command 'crictl ps -a' and it shows the 'etcd', 'apiserver' are not running:

# crictl ps -a
CONTAINER           IMAGE               CREATED              STATE               NAME                      ATTEMPT             POD ID
f556368cde2bb       a6ee862741f5c       About a minute ago   Exited              kube-apiserver            25047               e149dab807a6b
8225fbb23c5e7       9d8f85302c329       6 minutes ago        Exited              etcd                      17701               f6cb2952f4c8f

• Since 'apiserver' depends on 'etcd', check the logs of 'etcd' it shows below messages which indicates the issue is about certificate:

# crictl logs 8225fbb23c5e7
yyyy-mm-ddThh:mm:ss.Z stderr F {"level":"warn","ts":"yyyy-mm-ddThh:mm:ss.Z", "caller":"embed/config_logging.go:169", "msg":"rejected connection","remote-addr":"<SPVM-IP-ADDRESS>:47780","server-name":"","error":"remote error: tls: bad certificate"}

Cause

The certificates for the Supervisor Cluster have expired, which resulted in the apiserver not functioning as expected.

Resolution

Check the certificates and renew certificates using Replace vSphere with Tanzu Supervisor Certificates

Additional Information

Master vSphere Supervisor Certificate Guide