vmware-system-tmc namespace looks stuck in Terminating state when you Remove a Cluster from Your Organization

Products

• VMware Tanzu Mission Control
• Tanzu Mission Control
• Tanzu Mission Control Prepaid Commitment Plan per Core

Issue/Introduction

Scenario:

• You attempt to Remove a Cluster from Your Organization from VMware Tanzu Mission Control
  
  
• This process requires deleting the vmware-system-tmc namespace.
  
  
• When you attempt to delete the vmware-system-tmc namespace, the command hangs and/or the namespace stays in Terminating state
  
  $ kubectl delete namespace vmware-system-tmc
  
  

$ kubectl get namespace
NAME              STATUS      AGE
vmware-system-tmc Terminating 478d

 

• Attempts to delete the vmware-system-tmc forcefully (using --force option) is not successful

 

Errors:

 

• If you describe vmware-system-tmc namespace you see errors similar to the following:

Discovery failed for some groups, 2 failing: unable to retrieve the complete list of server APIs: identity.concierge.pinniped.tmc.cloud.vmware.com/v1alpha1: the server is currently unable to handle the request, login.concierge.pinniped.tmc.cloud.vmware.com/v1alpha1: the server is currently unable to handle the request"

 

State:

• The cluster may report as Disconnected Unknown in the Tanzu Mission Control console web UI

Environment

Tanzu Mission Control

vSphere IaaS Control Plane (formally known as vSphere with Tanzu)

Cause

• In this scenario, the vmware-system-user account had expired for the workload cluster.
  
  
• First, the expired system account had to be resolved.

Resolution

As a customer operator:

• Follow the below KB to resolve the expired account. vmware-system-user in this example:
  
  
  • • KB Article ID 319375: vmware-system-user account expired on TKGS Guest Cluster node
      
       
• Check and validate that no objects exist in TMC namespace before forceful delete:
  
  

  kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n vmware-system-tmc

• If objects still exist, open a Tanzu Support request case before attempting any further forceful deletions of any objects.