Troubleshooting Kubectl vSphere Contexts

Products

Tanzu Kubernetes Runtime VMware vSphere Kubernetes Service

Issue/Introduction

This KB is written to provide steps on how to troubleshoot issues with contexts in kubectl vsphere for the vSphere Kubernetes Service (VKS) and vSphere Supervisor product.

Environment

vSphere Supervisor

vSphere Kubernetes Service (VKS)

vCenter 8.X

Cause

Permissions and privileges are assigned based on SSO Permissions at the namespace level.

The list of available contexts is automatically populated based on the SSO Permissions assigned at the time of the login.

Permission changes will apply after the next kubectl vsphere login.

See the following vSphere 8 documentation for details:

Resolution

In the VKS product, kubectl vsphere login authenticates against the Supervisor cluster and should use the server flag for the Supervisor cluster's Control Plane Node Address. This can be found under Workload Management or Supervisor Management (in vCenter 9) and navigating to the Supervisors table tab.

To login to the Supervisor cluster context, use the following command:

kubectl vsphere login --server <Supervisor Control Plane Node Address> --vsphere-username <SSO username>

For workload/VKS cluster contexts, specify the namespace and cluster name, similar to the following command:

kubectl vsphere login --server <Supervisor Control Plane Node Address> --vsphere-username <SSO username> --tanzu-kubernetes-cluster-namespace <namespace> --tanzu-kubernetes-cluster-name <cluster>

After a successful login using kubectl vsphere login, details on the list of available contexts can be viewed with the below command:

kubectl config get-contexts

The CLUSTER column will display the corresponding IP address related to the context and can be used to validate the context accordingly.
For the Supervisor cluster context, the CLUSTER IP address matches the Control Plane Node Address noted above.
For any workload/VKS cluster context, the Control Plane Endpoint for the cluster matches the CLUSTER IP address.
- In the vSphere web UI, this can be found under the namespace for the cluster's Compute tab under VMware Resources - Tanzu Kubernetes clusters.
- From the Supervisor cluster context, you can run the following command to retrieve the expected IP for that cluster:
```
kubectl describe cluster -n <namespace> <cluster> | grep -i endpoint -A2
```

See the below table for error messages with recommendations related to context and SSO permissions:

Error Message or Issue	Explanation and Recommendations
FATA[YYYY-MM-DD HH:MM:SS.sss] Error while getting list of workloads: invalid or missing credentials	The login failed due to an incorrect username-password combination. Verify the correct SSO username and password.
Error from server (Forbidden): nodes is forbidden: User "sso:<SSO user>@<SSO domain>" cannot list resource "nodes" in API group "" at the cluster scope	This error message indicates that the SSO user was not assigned permissions for the current context. Specifically this error message states that the SSO user does not have permissions for querying the nodes object. Verify the current context and that the SSO user has appropriate permissions for the desired context. Confirm that the current context is the desired context based on the corresponding CLUSTER IP from "kubectl config get-contexts" For other objects than nodes, run the kubectl command against a specific namespace (-n <namespace>), instead of using -A for all namespaces. By default, the Supervisor cluster context limits permissions for Administrator, preventing commands from being run on system namespaces. As a result, this error will appear when running commands across all namespaces with the -A flag.
Logged into the wrong context but the context name should be correct.	Although the context name may match what you are looking for, each context is associated with a CLUSTER IP address. The context you are currently connected into may be pointed to a different CLUSTER IP address than desired. Run "kubectl config get-contexts" and compare the CLUSTER IP address to the desired context's Control Plane Address as per the above. If the desired context is not listed: -For workload/VKS cluster contexts, use kubectl vsphere login with the --tanzu-kubernetes-cluster flags to specify the desired context. -Verify that the SSO user has appropriate permissions for the desired context. There is a known issue where using the same name for a cluster and its namespace can result in context name confusion, or only one context instead of the expected two contexts of one context for the namespace and one context for the cluster. It is best practice to use different names for a cluster and its namespace.
The context I'm looking for is missing.	Your SSO user may not have appropriate permissions for the desired context or you need to specify the desired context when logging in. Confirm that your SSO user has the correct permissions for the desired context. For workload/VKS cluster contexts, use kubectl vsphere login with the --tanzu-kubernetes-cluster flags to specify the desired context. There is a known issue where using the same name for a cluster and its namespace can result in context name confusion, or only one context instead of the expected two contexts of one context for the namespace and one context for the cluster. It is best practice to use different names for a cluster and its namespace.

Additional Information

In vSphere Cloud Foundation 9 (VCF9), kubectl vsphere login is replaced with the VCF CLI.

Context Commands

kubectl vsphere login --server <Supervisor Control Plane Node Address>

```
--vsphere-username <SSO username>
```

--tanzu-kubernetes-cluster-namespace <namespace> --tanzu-kubernetes-cluster-name <cluster>

```
--insecure-skip-tls-verify
```
```
--verbose #
```

```
kubectl config current-context
```
- Output the current context that the SSO user is logged into.
```
kubectl config get-contexts
```
- List all contexts available that the logged in SSO user has permissions regarding.

kubectl config use-context <context name>

Switch to the given context.

Japanese version: Kubectl vSphere コンテキストのトラブルシューティング