Force deleting vks cluster by removing the finalizer can cause creating the cluster with the same name to fail

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

 

• The creating a VKS Cluster, with the same name as a previous deleted cluster, tasks is not proceeding.
• The are no machines or VM kubernetes objects created for the new created cluster, and there are only vspheremachine kubernetes objects.
  
  # kubect get ma,vspheremachine,vm -n cluster-ns | grep cluster-name
  

cluster-ns        vspheremachine.vmware.infrastructure.cluster.x-k8s.io/cluster-name-<ID>                  domain-<ID>   vsphere://<UUID>   <IP>
cluster-ns        vspheremachine.vmware.infrastructure.cluster.x-k8s.io/cluster-name-<ID>                  domain-<ID>   vsphere://<UUID>   <IP>
cluster-ns        vspheremachine.vmware.infrastructure.cluster.x-k8s.io/cluster-name-<ID>                  domain-<ID>   vsphere://<UUID>   <IP>
cluster-ns        vspheremachine.vmware.infrastructure.cluster.x-k8s.io/cluster-name-<ID>                  domain-<ID>   vsphere://<UUID>   <IP>
cluster-ns        vspheremachine.vmware.infrastructure.cluster.x-k8s.io/cluster-name-worker-<ID>                 vsphere://<UUID>   <IP>
cluster-ns        vspheremachine.vmware.infrastructure.cluster.x-k8s.io/cluster-name-worker-<ID>                 vsphere://<UUID>   <IP>
cluster-ns        vspheremachine.vmware.infrastructure.cluster.x-k8s.io/cluster-name-worker-<ID>                 vsphere://<UUID>   <IP>
cluster-ns        vspheremachine.vmware.infrastructure.cluster.x-k8s.io/cluster-name-worker-<ID>                 vsphere://<UUID>   <IP>
cluster-ns        vspheremachine.vmware.infrastructure.cluster.x-k8s.io/cluster-name-worker-<ID>                 vsphere://<UUID>   <IP>
cluster-ns        vspheremachine.vmware.infrastructure.cluster.x-k8s.io/cluster-name-worker-<ID>                 vsphere://<UUID>   <IP>


• There are no endpoints IP address assigned to the clsuter.

  NAMESPACE                 NAME                     ENDPOINTS                                                     AGE
  cluster-ns                cluster-name             <none>                                                        9m17s

   

• The machinedeployments k8s object is created and in ScalingUp state.
  
  Resource machinedeployments.cluster.x-k8s.io
  NAMESPACE          NAME                         CLUSTER        REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE       AGE     VERSION
  cluster-ns         cluster-name-worker-<ID>     cluster-name      4                                       ScalingUp   9m27s   v1.33.3+vmware.1-fips

   

• The kubeadmcontrolplanes k8s object is crated but with zero replicas.

  
  Resource kubeadmcontrolplanes.controlplane.cluster.x-k8s.io
NAMESPACE          NAME                  CLUSTER        INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE     VERSION
cluster-ns        cluster-name-<ID>     cluster-name                                                                                   9m33s   v1.33.3+vmware.1-fips
  
  

Environment

VMware vSphere Kubernetes Service

Cause

 

• The VKS cluster name machine is the same as an old VKS cluster that was force deleted by removing the finalizer which let there be a stale kubernetes objects for the old vks cluster that never got cleanup.
  
  EX:
  - There will be stale secrets that are showing old creation date (511d old)
  
  
  secret/cluster-name-gateway-api-package                          clusterbootstrap-secret               0      511d
secret/cluster-nameguest-cluster-auth-service-data-values       Opaque                                1      45m
secret/cluster-namekapp-controller-data-values                  Opaque                                2      45m
secret/clustr-name-kubeconfig                                   cluster.x-k8s.io/secret               1      511d
secret/cluster-name-token                                     Opaque                                2      45m
secret/cluster-name-metrics-server-package                       clusterbootstrap-secret               0      511d
secret/cluster-name-pinniped-package                             clusterbootstrap-secret               1      511d
secret/cluster-name-proxy                                        cluster.x-k8s.io/secret               2      45m
secret/cluster-name-sa                                           cluster.x-k8s.io/secret               2      45m
secret/cluster-name-secretgen-controller-package                 clusterbootstrap-secret               1      511d

Resolution

• To unblock: Create a new cluster using a unique new name.
• To clean up the stale Kubernetes objects from the deleted vks cluster: 
  
  • Delete the failed created cluster.
    • Delete a TKG Cluster Using Kubectl or Using the Tanzu CLI vSphere 8.0
    • Delete a VKS Cluster Using Kubectl or Using VCF CLI VCF 9.0
  • Ran the following commands to get all stale cluster k8s objects that didn't get  removed when the old VKS cluster was force deleted by removing the finalizer :
    # kubectl get cluster-api -n cluster-ns | grep cluster-name
# kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --ignore-not-found --show-kind -A | grep cluster-name
    # kubectl get secret -n cluster-ns | grep  cluster-name
    
    Note: The commands need to ran from the supervisor cluster context.
    
    
  • Collect a supervisor cluster logs 
  • Open a support case with Broadcom  and upload the collected logs and output to the case.