Supervisor NSX cleanup script fails due to a locked wcp-cluster-user

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Deactivating a vSphere Supervisor gets stuck with a Config Status of Removing
• Selecting the vSphere Client menu > Workload Management > Supervisors shows the supervisor's Config Status as Removing (view)
• Selecting the (view) link in shows the following status:

Supervisor deleted
Cleanup requests to NSX Manager failed.

• The vCenter server log /var/log/vmware/wcp/wcpsvc.log shows the following error:

<TIMESTAMP> error wcp [kubelifecycle/cluster_network.go:229] [opID=<OP_ID>] Received error cleaning NCP-created resources for cluster domain-<CLUSTER_ID> on NSX Managers: <NSX_MANAGER>:443. Err: exit status 1
nsx_policy_cleanup stdout:
nsx_policy_cleanup stderr: Traceback (most recent call last):
  File "/usr/lib/vmware-wcp/nsx_policy_cleanup.py", line 2051, in <module>
    nsx_client = NSXClient(host=options.mgr_ip,
  File "/usr/lib/vmware-wcp/nsx_policy_cleanup.py", line 204, in __init__
    self.header.update(provider.get_header_value())
  File "/usr/lib/vmware-wcp/jwt_session.py", line 429, in get_header_value
    token_value = self.get_token()
  File "/usr/lib/vmware-wcp/jwt_session.py", line 413, in get_token
    exchange_for_jwt()
  File "/usr/lib/vmware-wcp/jwt_session.py", line 248, in exchange_for_jwt
    saml_hok = self._sts_session.get_or_renew_hok_saml()
  File "/usr/lib/vmware-wcp/jwt_session.py", line 388, in get_or_renew_hok_saml
    hok_token = self.sts_auth.perform_request(
  File "/usr/lib/vmware/site-packages/pyVim/sso.py", line 277, in perform_request
    raise SoapException(fault, *parsed_fault)
pyVim.sso.SoapException: SoapException:
faultcode: ns0:FailedAuthentication
faultstring: The account of the user trying to authenticate is locked. :: The account of the user trying to authenticate is locked. :: User account locked: {Name: wcp-cluster-user-<UUID>, Domain: vsphere.local}
faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>The account of the user trying to authenticate is locked. :: The account of the user trying to authenticate is locked. :: User account locked: {Name: wcp-cluster-user-<UUID>, Domain: <DOMAIN></faultstring></S:Fault></S:Body></S:Envelope>

• From vCenter, running /usr/lib/vmware-vmafd/bin/dir-cli user find-by-name --account wcp-cluster-user* --level 2 shows the wcp-cluster-user account is locked out:

Enter password for [email protected]:
Account: wcp-cluster-user*
UPN: wcp-cluster-user-<UUID>@VSPHERE.LOCAL
Account disabled: FALSE
Account locked: TRUE
Password never expires: TRUE
Password expired: FALSE
Password expiry: N/A

Resolution

Reference wcp-cluster-user-domain User account password unlock and reset procedure