"Break-Glass" (Emergency Access) Approach for VMware vSphere Kubernetes (VKS)

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Users require a documented method to maintain administrative access to Tanzu Kubernetes (TKGS) cluster nodes and control plane components in the event of Identity Provider (IDP) or authentication service failures.

Environment

• VMware vSphere Kubernetes Service (VKS)
• VMware Tanzu Kubernetes Grid (TKGS)
• vSphere with Tanzu

Cause

Standard authentication workflows typically depend on external identity providers. During an IDP outage, administrators need a local, out-of-band mechanism to perform cluster maintenance or emergency recovery.

Resolution

To maintain access during authentication failures, use the local vmware-system-user account. This account exists on all guest cluster nodes and provides administrative access via sudo. Credentials (SSH private key or plaintext password) can be retrieved directly from the Supervisor Cluster.

Procedure to Retrieve Credentials

1. Log in to the Supervisor Cluster using a method that does not depend on the failed IDP (e.g., direct vCenter SSH access).
2. Identify the Target Cluster and Namespace:

   kubectl get clusters -A

3. Retrieve the SSH Private Key:

   kubectl get secret <cluster-name>-ssh -o jsonpath='{.data.ssh-privatekey}' -n <namespace> | base64 -d > /tmp/break-glass-keychmod 400 /tmp/break-glass-key

4. Retrieve the Plaintext Password (Optional):

   kubectl get secret <cluster-name>-ssh-password -o jsonpath='{.data.ssh-passwordkey}' -n <namespace> | base64 -d

5. Access the Cluster Node:

   ssh vmware-system-user@<node-ip> -i /tmp/break-glass-key

6. Switch to Root: Once logged in, use sudo -i to gain full administrative access.

Additional Information

Accessing vSphere with Tanzu workload clusters using SSH