vSphere Namespace stuck in "Terminating" state, 'vspheremachine' and 'persistentvolumeclaim' resources remain in same namespace

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• A vSphere Namespace is stuck in "Terminating" state for extend period of time
• Only either "vspheremachine" and/or "persistentvolumeclaim" resources remain in cluster, and no other resources:
  

  # kubectl api-resources --verbs=list --namespaced -o name --no-headers | xargs -n 1 kubectl get --show-kind --ignore-not-found --no-headers -n <vsphere-namespace> | awk -F'/' '{ print $1 }' | sort | uniq -c
  Warning: run.tanzu.vmware.com/tanzukubernetescluster is deprecated. It is succeeded by cluster.x-k8s.io/cluster. Refer to documentation.
    50 vspheremachine.vmware.infrastructure.cluster.x-k8s.io
     2 persistentvolumeclaim

Environment

vSphere Kubernetes Service

Cause

The exact root cause is still being under investigation.

Resolution

If this issue is experienced, please contact Broadcom Support for further analysis. If this issue has started recently, please collect a WCP Support Log Bundle from the affected Supervisor accordingly.