Guest cluster stuck in upgrade state with 4 Control Plane nodes in Ready state.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• During an upgrade of the Guest cluster, all newly created nodes are in Ready state. 
• The reconciliation loop fails to terminate and decommission the legacy node. 
• The etcd and kube-apiserver pods are in a non-functional state on the new node.
• There is a storage exhaustion on the legacy node. 

Environment

vSphere Kubernetes Service

Cause

 The disk pressure on the legacy node prevents the execution of node drainage required for deletion.

Resolution

• Storage Remediation: Perform a manual cleanup of the filesystem on the legacy node to alleviate disk pressure.
• Service Restoration: Restart the kubelet service on the new node to trigger a re-pull of container images and re-initialize the control plane components.
  • systemctl restart kubelet