TKGM Cluster creation succeeds but CSI pods stuck in CrashLoopBackOff

Products

• Tanzu Kubernetes Grid
• Tanzu Kubernetes Runtime
• VMware Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

After successfully creating a TKGM cluster, all CSI pods are in a CrashLoopBackOff State. 

Environment

• A Tanzu Kubernetes Grid multicloud environment with the following:
  
  

  1. kubectl get packages -A 

     • shows that the package with the package name "vsphere.csi.tanzu.vmware.com" is in a reconcile failed state.

  2. kubectl logs <VSPHERE-CSI-CONTROLLER-NAME> -n <NAMESPACE> -c vsphere-csi-controller 

     • shows the error "username <USERNAME> in vSphere config secret is invalid, make sure that username is a fully qualified domain name" 

Cause

The username is either configured with no domain, or the domain is placed before the username with only a single backslash. For example, the incorrectly configured user = "Domain\userName" or user = "userName" instead of the correctly configured user = "userName@Domain" or user = "Domain\\userName". If the domain name for the active directory user isn't specified, the vSphere Container Storage Plug-in will not function properly. 

Documentation on this requirement: Required Fields

Resolution

Follow the Update Credentials section of this Documentation: Update Credentials

To update the vSphere credentials used by the current standalone management cluster and by all of its workload clusters, use the command:

tanzu mc credentials update --cascading 

Run to log in to the management cluster that you are updating:

tanzu context use <MGMT-CLUSTER> 

 You can pass values to the following command options or let the CLI prompt you for them by running:

tanzu mc credentials update

--vsphere-user: Name for the vSphere account.
--vsphere-password: Password the vSphere account.
--vsphere-thumbprint: TLS thumbprint of the vCenter Server instance.

Additional Information

This can take up to 5 minutes for the cluster to take the change