Unable to deploy vsphere-csi addon due to vsphere-csi pod on the Cluster stuck in a CrashLoopBackOff state.

Products

• VMware Telco Cloud Automation

Issue/Introduction

When listing the CSI pod on the Cluster, the csi pods are stuck in CrashLoopBackOff state.

kubectl get pods -A | egrep "NAME|csi"
NAMESPACE       NAME                               READY   STATUS               RESTARTS   AGE
kube-system     vsphere-csi-controller-<ID>     5/7     CrashLoopBackOff   24      10m
kube-system     vsphere-csi-node-<ID>     2/3     CrashLoopBackOff   10      5m 
kube-system     vsphere-csi-node-<ID>     2/3     CrashLoopBackOff   10      1m

vsphere-csi-controller log: Found in /var/log/pods/vmware-system-csi_vsphere-csi-controller-<ID>/vsphere-csi-controller/#.log

failed to create govmomi client with err: ServerFaultCode: Cannot complete login due to an incorrect user name or password.
failed to connect to VirtualCenter host: \"vc.fqdn.com\", Err: ServerFaultCode: Cannot complete login due to an incorrect user name or password.

vsphere-syncer log: Found in /var/log/pods/vmware-system-csi_vsphere-csi-controller-<ID>/vsphere-syncer/#.log

   failed to create govmomi client with err: ServerFaultCode: Cannot complete login due to an incorrect user name or password.
    Cannot connect to vCenter with err: ServerFaultCode: Cannot complete login due to an incorrect user name or password.

 

Environment

2.x

Cause

CSI secret on the Cluster does not match the password configured for the user in the vCenter. 

Resolution

Follow the below steps:

1. Login to the Cluster with the capv user.
   Check password managed by vsphere-config-secret:
   

   kubectl get secrets vsphere-config-secret -n kube-system -o jsonpath='{.data.csi.vsphere\.conf}' | base64 -d

2.  Verify that the password configured for the SSO user is correct in the vsphere-config-secret or not. If not then modify the password, enter the entire command. Take reference from below example. Change the fields according to the environment specific variables:
   

   # cat <[Global]
   [Global]
   insecure-flag = true
   cluster-id = kube-system/cluster-01
   [VirtualCenter "vc.fqdn.com"]
   user = "[email protected]"
   password = ""
   datacenters = "DC"
   insecure-flag = true
   [Network)
   public-network = “/######/network/######/SEG-######”
   EOF

3.  This will modify the secret into base64 and will output the hash so you can enter it into the data.csi.vsphere.conf.
   
   
4. Run the following to edit the secret:
   

   kubectl edit secrets vsphere-config-secret -n kube-system

5. Delete the hash after csi-vsphere.conf and paste the new one you created from step 3.
   
   
6. Use :wq to write and quit the file, which will save the new secret.
   
   
7. Delete the csi pod to recreate it and instantiate the new secret:
   

   kubectl delete pod <csi-pod-name> -n kube-system
   

   Pods should now be up and running.
   
   Note: If the issue still persists then

   • Remove the vsphere-csi addon
   • Make sure that the secret is also deleted. If not then delete the secret manually
   • Recreate the vsphere-csi addon with correct SSO credentials