Control plane node stuck in 'Creating' state during TKR upgrade due to kube-vip Pod failure in TKGm

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

During a Tanzu Kubernetes Grid multi-cloud (TKGm) upgrade (e.g., v2.5.0 to v2.5.4) or a Tanzu Kubernetes Release (TKR) upgrade (e.g., v1.30.13 to v1.31.9), the following symptoms occur:

• A control plane node remains stuck in a Creating or Provisioning state indefinitely.
• The cluster Virtual IP (VIP) is unreachable, resulting in Unable to connect to the server errors from kubectl.
• Worker nodes transition to a Not Ready state.
• The 'KubeadmControlPlane' (KCP) controller in the management cluster stalls and fails to provision replacement nodes.

As part of recovery, attempted to delete the nodes from vCenter and remove finalizers from the machine objects but these actions didn't restore the upgrade process

Environment

2.5.4

Cause

The primary cause is a failure in the kube-vip pod's lifecycle during the rolling upgrade process. If the VIP is not correctly allocated to an active control plane node, the management plane cannot communicate with the workload cluster's API server. This communication break prevents the KCP controller from reconciling the cluster state and progressing with the upgrade.

Resolution

To restore cluster connectivity and allow the upgrade to resume, follow these steps:

1. Restore API Connectivity

   • Identify the active control plane nodes and forcefully restart the `kube-vip` pods to trigger a new VIP election. Ensure you have SSH access to the control plane nodes before proceeding.

     1. SSH into each active control plane node.

     2. Force delete the `kube-vip` pod:
           kubectl delete pod -n kube-system <kube-vip-pod-name> --force --grace-period=0

     3. Verify the VIP has been assigned to a node:
           ip addr show eth0

2. Reconcile the KCP Controller

kubectl annotate kcp -n <namespace> <cluster-name>-control-plane  "cluster.x-k8s.io/force-reconcile=$(date +%s)" --overwrite

 This successfully reassigned the VIP to one of the nodes, stabilized the control plane, and allowed the third control plane node to be recreated and join the cluster.