PVC resize operations fail due to wcp-storage-user account lockout and Envoy overload

Products

• VMware vSphere Kubernetes Service
• VMware vCenter Server

Issue/Introduction

In vSphere with Tanzu environments, PersistentVolumeClaim (PVC) resize operations fail due to a service account lockout. This condition occurs when the Supervisor CSI controller is unable to authenticate with vCenter Server.

Symptoms:

• Supervisor csi-controller and csi-resizer logs contain: err=ServerFaultCode: Cannot complete login due to an incorrect user name or password.
• vCenter server logvmdird.log contains lockout events for the storage service principal associated with the supervisor: WARNING: LoginBlocked DN (cn=wcp-storage-user-...), error (9241)(Account access blocked).
• vCenter websso.log and vmware-identity-sts-default.logcontain envoy overloaded errors.
• Dir-cli confirms the account is locked.

  /usr/lib/vmware-vmafd/bin/dir-cli user find-by-name --account wcp-storage-user-#######-####-####-####-########-########-####-####-####-######## --level 2
  Enter password for [email protected]:
  Account: wcp-storage-user-#######-####-####-####-########-########-####-####-####-########
  UPN: wcp-storage-user-#######-####-####-####-########-########-####-####-####-########@VSPHERE.LOCAL
  Account disabled: FALSE
  Account locked: TRUE
  Password never expires: FALSE
  Password expired: FALSE
  Password expiry: 89 day(s) 23 hour(s) 53 minute(s) 33 second(s)

Environment

Product: VMware Cloud Foundation / vSphere with Tanzu

Component: Supervisor Cluster, CSI Driver, VMware vSphere Kubernetes Service (VKS)

Version: vCenter Server 8.0 Update 3g (build 24155106) and earlier

Cause

Envoy-sidecar heap memory exhaustion prevents wcpsvc from rotating the wcp-storage-user password every 12 hours.
This leads to account lockouts and CSI operation failures.

Resolution

Fixed in release vCenter Server 8.0 U3h and higher. See Download Broadcom products, patches and software for steps to download this release.

Workaround:

1. Increase envoy-sidecar heap memory per KB 384498.
2. SSH to the vCenter and stop the wcp service
   

   service-control --stop wcp

3. Connect to the Supervisor cluster and validate the current number of csi-controller replicas
   

   kubectl get deployment vsphere-csi-controller -n vmware-system-csi

4. Scale down the Supervisor csi-controller pods to 0 to prevent the controller from attempting to authenticate using the wcp-storage-user
   

   kubectl scale deployment vsphere-csi-controller -n vmware-system-csi --replicas=0

5. SSH to the vCenter and unlock the wcp-storage-user account
   

   /opt/likewise/bin/ldapmodify -x -D cn=Administrator,cn=Users,dc=vsphere,dc=local -W <<EOF
   dn: cn=wcp-storage-user-#######-####-####-####-########-########-####-####-####-########,cn=serviceprincipals,dc=vsphere,dc=local
   changetype: modify
   replace: userAccountControl
   userAccountControl: 0
   EOF

6. Confirm the account is unlocked
   

   /usr/lib/vmware-vmafd/bin/dir-cli user find-by-name --account wcp-storage-user-#######-####-####-####-########-########-####-####-####-######## --level 2

   The output will show the account locked value is FALSE

   Enter password for [email protected]:
   Account: wcp-storage-user-#######-####-####-####-########-########-####-####-####-########
   UPN: wcp-storage-user-#######-####-####-####-########-########-####-####-####-########@VSPHERE.LOCAL
   Account disabled: FALSE
   Account locked: FALSE
   Password never expires: FALSE
   Password expired: FALSE
   Password expiry: 89 day(s) 23 hour(s) 53 minute(s) 33 second(s)

7. Start the vCenter wcp service. This will trigger a password refresh for the wcp-storage-user. 
   

   service-control --start wcp

8. Scale the csi controller deployment back to the original  number recorded in step 3.
   

   kubectl scale deployment vsphere-csi-controller -n vmware-system-csi --replicas=03

9. Monitor the vmdird.log and ensure the wcp-storage-user account does not generate any new authentication failure errors or warnings