wcp-cluster-user-domain User account password unlock and reset procedure
search cancel

wcp-cluster-user-domain User account password unlock and reset procedure

book

Article ID: 305321

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vSphere Kubernetes Service

Issue/Introduction

vSphere with Tanzu uses the wcp-cluster-user-domain user account to allow the wcp-schedext pod to authenticate with vCenter in order to translate scheduler operations into DRS.

A locked wcp-cluster-user-domain account or an out-of-sync password prevents vSphere Supervisor Clusters from synchronizing the wcp-schedext pod against DRS for scheduling decisions, forcing Supervisor Cluster pods to remain in a Pending state

  • Similar errors to the following are found in the Supervisor wcp-schedext pod logs: 
YYYY-MM-DDTHH:MM:SS stderr F YYYY-MM-DDTHH:MM:SS error schedext [opID=cfgMapUpdate-40a0] Could not login to vCenter. Error: ServerFaultCode: Cannot complete login due to an incorrect user name or password.
  • In vCenter server, the /var/log/vmware/vmdird/vmdird-syslog.log and vmdird.log may show similar errors to;
YYYY-MM-DDTHH:MM:SS err vmdird  t@139774299973376: SASLSessionStep: sasl error (-13)(SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)
YYYY-MM-DDTHH:MM:SS warning vmdird  t@139774299973376: Lockout policy check - account lockout. (cn=wcp-cluster-user-domain-<ClusterID>-<VC_MachineID>,cn=serviceprincipals,dc=domain,dc=local)

 

The same errors may effect other wcp service accounts, such as wcp-storage-management or workload-storage-user. 

Environment

VMware vSphere 7.0
VMware vSphere 8.0

Cause

This condition occurs most commonly after Supervisor Cluster certificates expire.

If the wcp-cluster-user-domain account is locked, or if the password is out of sync, vSphere Supervisor Clusters will not be able to synchronize the wcp-schedext pod against DRS for scheduling decisions, leading to Supervisor Cluster pods hanging in Pending state.

Resolution

Automatic WCP user account password sync occurs every 12 hours.
Manually restarting the wcp service in the vCenter initiates an automatic password reset for all wcp solution users, and can be done through the VAMI, or by running the following command while connected to the vCenter through a SSH:
service-control --restart wcp
 
If the automatic rotation is not functioning, apply the below workaround. 

Workaround:

Check the affected WCP solution user account lock status:

  • From vCenter SSH, check /var/log/vmware/wcp/wcpsvc.log for the wcp-cluster-user-domain account ID.

    • Example of wcp-cluster-user-domain user ID: wcp-cluster-user-domain-c#-#####-####-####-####-#########@domain.local
    • The c# is the ClusterID on which WCP was built. The #####-####-####-####-######### is the vCenter MachineID.

  • From Supervisor VM, check wcp-schedext pod logs to see if they're reporting login failures: 

kubectl logs -n kube-system kube-scheduler-<POD_ID> -c wcp-schedext | less

  • Check /var/log/vmware/vmdird/vmdird-syslog.log on the vCenter to determine if the account is locked:

YYYY-MM-DDTHH:MM:SS warning vmdird  t@140502791870208: LoginBlocked DN (cn=wcp-cluster-user-domain-<ClusterID>-<VC_MachineID>,cn=serviceprincipals,dc=domain,dc=local), error (9241)(Account access blocked)
YYYY-MM-DDTHH:MM:SS err vmdird  t@139774291580672: VmDirSendLdapResult: Request (Bind), Error (LDAP_INVALID_CREDENTIALS(49)), Message ((49)(SASL step failed.)), (0) socket (127.0.0.1)
YYYY-MM-DDTHH:MM:SS err vmdird  t@139774291580672: Bind Request Failed (127.0.0.1) error 49: Protocol version: 3, Bind DN: "CN=wcp-cluster-user-domain-<ClusterID>-<VC_MachineID>,cn=ServicePrincipals,dc=domain,dc=local", Method: SASL 

  • Check the user status on the vCenter by using the following command:

/usr/lib/vmware-vmafd/bin/dir-cli user find-by-name --account wcp-cluster-user-domain-c#-#####-####-####-####-######### --level 2
 
The output will be similar to:

Account: wcp-cluster-user-domain-c#-#####-####-####-####-#########
UPN: wcp-cluster-user-domain-c#-#####-####-####-####-#########@domain.local
Account disabled: FALSE
Account locked: TRUE
Password never expires: FALSE
Password expired: FALSE
Password expiry: 9998 day(s) 19 hour(s) 57 minute(s) 58 second(s)

  • If the account is locked, use the command below to unlock
    • This command executes everything between <<EOF and the final line EOF):
       
      /opt/likewise/bin/ldapmodify -x -D cn=Administrator,cn=Users,dc=vsphere,dc=local -W <<EOF
      dn: CN=wcp-cluster-user-domain-c#-#####-####-####-####-#########,CN=ServicePrincipals,dc=domain,dc=local
      changetype: modify
      replace: userAccountControl
      userAccountControl: 0
      EOF

 

The steps above can be updated for other affected WCP solution users. 
If issue persists after attempting to unlock, please open a case with Broadcom Support. 
Creating and managing Broadcom cases

Additional Information

See the following KBs for additional WCP user troubleshooting guidance:
vSphere with Tanzu WCP service authentication failure for wcp-storage-user due to account lockout 
FailedLoginAttempts SupervisorControlPlaneVM Login Issues for wcp-vmimageserviceop-user
User wcp-appplatform-user account password unlock and reset procedure
 


Powered by Wolken Software