vmware-system-tmc namespace looks stuck in Terminating state when you remove a Cluster from your organization

Products

• VMware Tanzu Mission Control
• Tanzu Mission Control
• Tanzu Mission Control Prepaid Commitment Plan per Core

Issue/Introduction

• You attempt to Remove a Cluster from Your Organization from VMware Tanzu Mission Control
  
  
• This process requires deleting the vmware-system-tmc namespace.
  
  
• When you attempt to delete the vmware-system-tmc namespace, the command hangs and/or the namespace stays in Terminating state
  
  $ kubectl delete namespace vmware-system-tmc
  
  

$ kubectl get namespace
NAME              STATUS      AGE
vmware-system-tmc Terminating 478d

• Attempts to delete the vmware-system-tmc forcefully (using --force option) is not successful

Errors:

• If you describe vmware-system-tmc namespace you see errors similar to the following:

Discovery failed for some groups, 2 failing: unable to retrieve the complete list of server APIs: identity.concierge.pinniped.tmc.cloud.vmware.com/v1alpha1: the server is currently unable to handle the request, login.concierge.pinniped.tmc.cloud.vmware.com/v1alpha1: the server is currently unable to handle the request"

State:

• The cluster may report as Disconnected Unknown in the Tanzu Mission Control console web UI

Environment

• Tanzu Mission Control
• vSphere IaaS Control Plane (formally known as vSphere with Tanzu)

Cause

• The vmware-system-user account has expired for the workload cluster.
• First, the expired system account has to be resolved.

Resolution

As a customer operator:

1. Follow the below KB to resolve the expired account. vmware-system-user in this example:
   
   
   • • KB Article ID 319375: vmware-system-user account expired on TKGS Guest Cluster node
        
2. Check and validate that no objects exist in TMC namespace before forceful delete:
   
   

   kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n vmware-system-tmc

3. If objects still exist, open a Tanzu Support request case before attempting any further forceful deletions of any objects.