Delete specific worker node when scale in vSphere with Tanzu guest cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

When scale in worker node of vSphere with Tanzu guest cluster (TKC), worker nodes will be deleted randomly. The article is to provide solution how to delete specific worker node.

Environment

VMware vSphere with Tanzu

Resolution

Important Note: Newer versions of VKS utilize the standard upstream Cluster API (cluster object), while older versions use the deprecated TanzuKubernetesCluster (tkc object) API. The steps below apply to both, but you must use the correct object type and YAML path for your specific cluster version when editing.

Follow the steps to delete specific worker node when scale in:

1. SSH one of Supervisor node with the KB Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VMs
   
   
2. Get the machine list of the guest cluster:
   
   kubectl get machine -n <namespace>
   
   
3. Annotate the worker node:
   
   kubectl annotate machine -n <namespace> <worker-machine-name-to-delete> "cluster.x-k8s.io/delete-machine"="yes"
   
   
4. Edit the cluster / tkc Object
   
   Edit the guest cluster object based on your cluster type (cluster for modern VKS clusters, or tkc for legacy clusters):
   
   For modern clusters: kubectl edit cluster -n <namespace> <cluster-name>
   
   For legacy TKC clusters: kubectl edit tkc -n <namespace> <tkc-name>
   
   
5. Scale in the workers by decreasing the replicas number:
   
   

   workers:
     machineDeployments:
     - class: node-pool
       metadata:
         annotations:
           run.tanzu.vmware.com/resolve-os-image: os-name=photon
       name: kubernetes-cluster-#####-#######
       replicas: 3 # <-- Decrease this number

   
   
   
6. Save the change.

Additional Information

Manually Scale a Cluster Using Kubectl