vSphere with Tanzu WCP service authentication failure for wcp-storage-user due to account lockout

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Issue where the WCP service fails to function correctly, matching the symptoms described in KB 420549. Restarting the WCP service does not resolve the issue. Upon reviewing the /var/log/vmware/vmdird/vmdird-syslog.log, authentication failures associated with the wcp-storage-user account are present.

Cause

The vsphere-csi-controller pod in crash loop backup holds a stale credential and is locking out the wcp-storage-user

Resolution

1. To obtain the full service account name ssh to the vCenter as root and run cat /var/log/vmware/vmdird/vmdird.log | grep wcp-storage-user-

   Example output: 
   
   YYYY-MM-DDTHH:MM:SS INFO: VmDirGetAccountUPN success for AccountUPN (wcp-storage-user-4ac####-####-####-####-########-########-####-####-####-######[email protected])

2. Confirm that the user ID from the output matches the account experiencing issues in the wcpsvc.log files.
3. In the vCenter command line verify the account state using the dir-cli command to ensure the account is locked and the password expiry is in the future. Note if the account is not locked, see KB 438065
   /usr/lib/vmware-vmafd/bin/dir-cli user find-by-name --account wcp-storage-user-<ID> --level 2

   Example Output:
   
   Enter password for [email protected]:
   
   Account: wcp-storage-user-4ac####-####-####-####-########-########-####-####-####-######c3
   
   UPN: wcp-storage-user-4ac####-####-####-####-########-########-####-####-####-######[email protected]
   
   Account disabled: FALSE
   
   Account locked: FALSE
   
   Password never expires: FALSE
   
   Password expired: FALSE
   
   Password expiry: 89 day(s) 23 hour(s) 53 minute(s) 33 second(s)

4. Queue up the unlock command in the next step so that it can be run immediately. Once you have the unlock command ready, bring down the CSI controller pods in crash loop back off with: 
   kubectl delete pod <vmware-csi-controller-id> -n vmware-system-csi 
5. Unlock the wcp-storage-user account while the pods are recreating.
   /opt/likewise/bin/ldapmodify -x -D cn=Administrator,cn=Users,dc=vsphere,dc=local -W <<EOF
   dn: cn=wcp-storage-user-4ac####-####-####-####-########-########-####-####-####-######c3,cn=serviceprincipals,dc=vsphere,dc=local
   changetype: modify
   replace: userAccountControl
   userAccountControl: 0
   EOF
6. Verify the pods com back into a running state.

Additional Information

FailedLoginAttempts SupervisorControlPlaneVM Login Issues for wcp-vmimageserviceop-user