vSphere with Tanzu TKC login fails with "Login failed: bad gateway"

Products

• VMware vSphere with Tanzu

Issue/Introduction

When try to login TKC in vSphere with Tanzu, fails with below messages:

# kubectl vsphere login --server <KUBERNETES-CONTROL-PLANE-IP-ADDRES> --tanzu-kubernetes-cluster-namespace <KUBERNETES-NAMESAPCE> --tanzu-kubernetes-cluster-name <TKC-NAME> -u <TKC-USER-NAME> 
Error Message:
KUBECTL_VSPHERE_PASSWORD environment variable is not set. Please enter the password below 
Password: 
ERRO[0006] Login failed: bad gateway 
ERRO[0006] Failed login to Tanzu Kubernetes cluster <TKC-NAME>: bad gateway 

Cause

The certificates of Supervisor cluster are expired that caused the apiserver was not working normally.

Resolution

Check the containers status in each SPVM with command 'crictl ps -a' and it will show the 'etcd', 'apiserver' are not running:

# crictl ps -a
CONTAINER           IMAGE               CREATED              STATE               NAME                      ATTEMPT             POD ID
f556368cde2bb       a6ee862741f5c       About a minute ago   Exited              kube-apiserver            25047               e149dab807a6b
8225fbb23c5e7       9d8f85302c329       6 minutes ago        Exited              etcd                      17701               f6cb2952f4c8f

Since 'apiserver' depends on 'etcd', check the logs of 'etcd' it will show below messages which indicates the issue is about certificate:

# crictl logs 8225fbb23c5e7
2024-05-06T05:53:16.820350763Z stderr F {"level":"warn","ts":"2024-05-06T05:53:16.820Z", "caller":"embed/config_logging.go:169", "msg":"rejected connection","remote-addr":"<SPVM-IP-ADDRESS>:47780","server-name":"","error":"remote error: tls: bad certificate"}

Check the certificates and renew certificates with Replace vSphere with Tanzu Guest Cluster Certificates

Additional Information

Main vSphere with Tanzu certificate page https://knowledge.broadcom.com/external/article?articleNumber=323421