vSphere Supervisor Namespace, Cluster or TKC stuck in Deleting Phase

Products

• VMware vSphere Kubernetes Service
• Tanzu Kubernetes Runtime
• VMware vSphere 7.0 with Tanzu
• vSphere with Tanzu

Issue/Introduction

When attempting to delete an object in a vSphere Supervisor environment, the object remains stuck in Deleting state.

This KB article will go over checks regarding if the deleting object is a Namespace, Cluster or TanzuKubernetesCluster (TKC) object.

• When viewing the namespace from the vSphere web UI, it shows as stuck in Terminating state.
• From the Tanzu Mission Control web UI, the affected workload cluster will show as Removing.
• While connected to the Supervisor cluster context, the namespace is Terminating and the underlying workload cluster objects are in deleting state.

 

For Workload Cluster nodes stuck deleting, please check the below KB article regarding PodDisruptionBudgets (PDBs):

vSphere Workload Cluster Node Stuck Deleting due to PodDisruptionBudget (PDB)

Environment

vSphere Supervisor 7.0

vSphere Supervisor 8.0

vSphere Supervisor 9.0

This issue can occur regarding of whether or not the affected resource is managed by Tanzu Mission Control (TMC).

Cause

Object deletions can get stuck when incorrect steps are performed for deletion. The deletion request remains stuck as it is waiting for existing underlying objects and dependencies to delete.

For example:

• Deleting the namespace from vSphere web UI before deleting the underlying components in the namespace.
• Node deletion from within the Workload Cluster context.
• Deleting a namespace through the CLI prior to deleting the underlying objects in the namespace.

 

Resolution

System object deletion without support's guidance can irrecoverably break the environment or render the environment as unsupported.

 

Any object that is stuck in deleting state can be described for more information.

• For Terminating namespaces, the describe will show a list of underlying objects that it is awaiting deletion under Conditions.
  • In the below example, the namespace stuck Terminating is waiting for underlying classy cluster objects to finish deleting, the actual resources remaining will vary by environment and deletion process:

  • kubectl describe ns <namespace>
    
    Status: Terminating
    Conditions:
       Type    Reason    Message     Status    LastTransitionTime
      ContentDeletionFailed    Failed to delete all resource types, # remaining: admission webhook "validation.clusterclass.cluster.x-k8s.io" denied the request: ClusterClass.cluster.x-k8s.io "<clusterclass name>" is forbidden: ClusterClass cannot be deleted because it is used by # Cluster(s).
      NamespaceContentRemaining      True      DAY, DD MON YYYY HH:MM:SS +ssss
            SomeResourcesRemain   Some resource are remaining: clusterbootstraps.run.tanzu.vmware.com has # resource instances, clusters.cluster.x-k8s.io has # resource instances, kubeadmcontrolplanes.controlplane.cluster.x-k8s.io has # resource instances, machines.cluster.x-k8s.io has # resource instances, secrets. has # resource instances
    NamespaceFinalizersRemaining     True     DAY, DD MON YYYY HH:MM:SS +ssss
          SomeFinalizersRemain   Some content in the namespace has finalizers remaining: cluster.cluster.x-k8s.io in # resource instances, kubeadmcontrolplanes.controlplane.cluster.x-k8s.io in # resource instances, machines.cluster.x-k8s.io in # resource instances, tkg.tanzu.vmware.com/addon in # resource instances
    
    

• For Workload Cluster nodes stuck deleting, please check the below KB article regarding PodDisruptionBudgets (PDBs):

  vSphere Workload Cluster Node Stuck Deleting due to PodDisruptionBudget (PDB)

 

Any further actions should be to investigate the noted underlying objects or dependencies for the stuck Deleting object, which are also likely stuck deleting/terminating.

IMPORTANT: It is not an appropriate practice to delete the finalizers of a deleting object.

Finalizers exist to prevent untimely or inappropriate deletion of objects which underlying resources are dependent upon.

Deletion of finalizers leads to stale and orphaned objects in the environment which the system can no longer interact with to clean up or manage.

 

Please contact VMware by Broadcom Technical Support for assistance on cleaning up stuck objects that are stuck Deleting or stuck Terminating.