Namespace stuck in Removing in vCenter

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

• The namespace is stuck in deleting and does not proceed after some time
• When describing the namespace we see in Conditions a message similar to:
  

  Some content in the namespace has finalizers remaining: cns.vmvare.com in 6 resource instances, cns.vmvare.com/pvc-protection in 6 resource instances, kubernetes.io/pvc-protection in 4 resource instances, vspheremachine.infrastructure.cluster.x-k8s.io in 7 resource instances

• Run the commands below to see if there are resources in the namespace: 
  

  kubectl get vspheremachines.vmware.infrastructure.cluster.x-k8s.io -n <namespace>
  
  NAME                                       ZONE                 PROVIDERID                                          IPADDR
  worker-name                                                       vsphere://xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx            ###.###.##.##
  
  
  
  
  kubectl get cnsnodevmattachments.cns.vmware.com -n <namespace>
  
  NAME                                                          AGE
  worker-name-xxxxxxx-xxxxx-xxxx-xxxxx-xxxxx-xxxx                 22h
  
  
  
  
  
  kubectl get pvc -n <namespace> 
  
  NAMESPACE   NAME                                                                STATUS        VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         VOLUMEATTRIBUTESCLASS   AGE
  <namespace>         persistentvolumeclaim/xxxxxx-xxxx-xxxx-xxxx--xxx               Terminating   pvc-xxxxxxxxxxx-xxxxx-xxxx-xxxxx-xxxxx   4884452Mi   RWO            tkg-storage-policy   <unset>                 54m
  

Note: The above commands will depend on your environment and what is left over. There could be more or less of these objects.

Environment

Tanzu Kubernetes Grid Service (TKGs) 7.x and 8.x

 

Cause

This is caused by a race condition where the needed components were deleted before the attached Persistent Volume Claims (PVC), and Persistent Volumes (PV) components could be removed.

Resolution

Open a Support Request and reference this KB