vSphere with Tanzu cluster stuck in Removing state with the message "info: NSX Resource cleanup is in progress."

Products

• VMware vSphere ESXi

Issue/Introduction

• Disabling vSphere with Tanzu from Workload Management -> Clusters page hangs forever with the message " info: NSX Resource cleanup is in progress."
  
  
• Verifying the logs in vCenter Server under /var/log/vmware/wcp/wcpsvc.log:
  
  YYY-MM-DDTHH:MM:SSZ debug wcp [opID=####-domain-c####] Looking for agencies matching 'vmware-vsc-apiserver' (prefixMatch: true) in set of 0
YYY-MM-DDTHH:MM:SSZ debug wcp [opID=####-domain-c####] Lock found for nsx_policy_cleanup script for cluster domain-c####:<ID>. Cleanup in progress, retry later. stdout: ERROR: Failed to create sessionID with endpoint https://<VC_FQDN>:443/rest/com/vmware/cis/session: Status 401: Error: {"type":"com.vmware.vapi.std.errors.unauthenticated","value":{"error_type":"UNAUTHENTICATED","messages":[{"args":[],"default_message":"Authentication required.","id":"com.vmware.vapi.endpoint.method.authentication.required"}],"challenge":"Basic realm=\"VAPI endpoint\",SIGN realm=####,service=\"VAPI endpoint\",sts=\"https://vcenter.example.com/sts/STSService/vsphere.local\""}}
. stderr: Traceback (most recent call last):
  File "/usr/lib/vmware-wcp/nsx_policy_cleanup.py", line 1588, in <module>
    all_res=options.all_res)
  File "/usr/lib/vmware-wcp/nsx_policy_cleanup.py", line 155, in __init__
    self.header.update(provider.get_header_value())
  File "/usr/lib/vmware-wcp/jwt_session.py", line 425, in get_header_value
    token_value = self.get_token()
  File "/usr/lib/vmware-wcp/jwt_session.py", line 408, in get_token
    jwt_resp, use_old_audience = self._tes_session. \
  File "/usr/lib/vmware-wcp/jwt_session.py", line 249, in exchange_for_jwt
    session_id = self._retrieve_vapi_session(saml_hok)
  File "/usr/lib/vmware-wcp/jwt_session.py", line 241, in _retrieve_vapi_session
    raise Exception
Exception
YYY-MM-DDTHH:MM:SSZ warning wcp [opID=####-domain-c####] NSX resource removal did not fully complete for cluster domain-####. Retrying. Err: NSX cleanup in progress. This operation is part of NSX cleanup and will be retried.

Environment

vSphere Kubernetes Service

Cause

This happens due to an uppercase letter in the vCenter hostname causing the JWT (java web token) to NSX to fail.

Resolution

This issue has been fixed in NSX Container Plugin(NCP) 3.0.1 release.

Workaround:

1. Create a backup of the script to be edited.
   cp /usr/lib/vmware-wcp/jwt_session.py /usr/lib/vmware-wcp/jwt_session.py.backup
   
   
2. Edit /usr/lib/vmware-wcp/jwt_session.py on VCSA, find line
   
   norm_req += "\n" + self._vc_endpoint
   and change it to:
   norm_req += "\n" + self._vc_endpoint.lower()
   
   
3. No need to restart anything. Wait for the components to clean up again or try to disable the cluster.