kubectl vsphere login to Supervisor Cluster or any VKS Cluster is failing with "Login failed: bad gateway"

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

1. All kubectl vsphere login attempts are unsuccessful with "bad gateway", whether targeting the Supervisor cluster or a VKS cluster.
   
   • Supervisor Cluster:
     

     • kubectl vsphere login --vsphere-username <SSO@username> --server=<Workload Management Supervisor Control Plane Node IP Address>

   • TKG Service Cluster:

     • kubectl vsphere login --server <Workload Management Supervisor Control Plane Node IP Address> --vsphere-username <SSO@username> --tanzu-kubernetes-cluster-namespace <namespace> --tanzu-kubernetes-cluster-name <clusterName>

     • Error Message:
       

       KUBECTL_VSPHERE_PASSWORD environment variable is not set. Please enter the password below 
       Password: 
       time="YYYY-MM-DD HH:MM:SS" level=error msg="Login failed: bad gateway"
       time="YYYY-MM-DD HH:MM:SS" level=error msg="Failed login to Tanzu Kubernetes cluster my-tkg-cluster: bad gateway"
       Error: exit status 1

2. Also following error message may be seen from one or more Supervisor control plane nodes when viewing the Config Status of the Supervisor under the Workload Management section of the vSphere Client:

   • Customized guest of Supervisor Control plane VM
     Configuration error
     System error occurred on Master node with identifier ################################. Details: Log forwarding sync update failed: Command '['/usr/bin/kubectl', '--kubeconfig', '/etc/kubernetes/admin.conf', 'get', 'configmap', 'fluentbit-config-system', '--namespace', 'vmware-system-logging', '--ignore-not-found=true', '-o', 'json']' returned non-zero exit status 1..

3. Check that root disk space is above 80% or full at 100%:
   

   df -h /dev/root
   
   Filesystem Size Used Avail Use% Mounted on
   /dev/root    ##G ##G   ##G 100% /

Environment

vSphere with Tanzu

Cause

Root disk usage was above 80% or at 100% full on one or more Supervisor Cluster Control Plane VMs in a vSphere Supervisor environment, leading to this issue.

Resolution

To clear the root disk usage, follow KB vSphere Supervisor Root Disk Space Full at 100% and vSphere Supervisor Disk Space Clean Up Scripts

Once the disk space is cleared the supervisor is reconfigured successfully and we will be allowed to login to Supervisor Cluster Context and any TKG Service Cluster Context using vSphere Kubernetes Kubectl vSphere Login.