Supervisor Cluster control plane nodes randomly restart due to incompatible spherelet version

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Supervisor Cluster control plane nodes experience random downtime, leading to instability in production applications. The Supervisor Cluster typically displays an unhealthy status caused by frequent restarts of the etcd and apiserver pods, which often indicate an incompatible spherelet deployment.
  
  
• Execute the following command to verify the pod status:: kubectl get pods -A | egrep "NAMESPACE|etcd|apiserver"

An unhealthy environment displays a high number of restarts for these components, as seen in the example below:

# kubectl get pods -A | egrep "NAMESPACE|etcd|apiserver"
NAMESPACE                         NAME                                                              READY   STATUS      RESTARTS        AGE
kube-system                       etcd-################################                             1/1     Running     13 (36m ago)    204d
kube-system                       etcd-################################                             1/1     Running     2 (32m ago)     204d
kube-system                       etcd-################################                             1/1     Running     29 (27m ago)    204d
kube-system                       kube-apiserver-################################                   1/1     Running     178 (36m ago)   204d
kube-system                       kube-apiserver-################################                   1/1     Running     232 (32m ago)   204d
kube-system                       kube-apiserver-################################                   1/1     Running     327 (27m ago)   204d

• Additionally, the /var/log/vmware/wcp/wcpsvc.log in the vCenter server indicate a Readiness Status of Unknown Condition for various nodes:

[YYYY-MM-DDTHH:MM:SS] debug wcp [kubelifecycle/node_reflector.go:124] Handling update for node 4200##########################d9 with status unknown. 
[YYYY-MM-DDTHH:MM:SS] debug wcp [kubelifecycle/node_reflector.go:124] Handling update for node 4200##########################d9 with status unknown.
[YYYY-MM-DDTHH:MM:SS] debug wcp [kubelifecycle/node_reflector.go:124] Handling update for node 4200##########################d9 with status unknown.
[YYYY-MM-DDTHH:MM:SS] debug wcp [kubelifecycle/node_reflector.go:124] Handling update for node 4200##########################20 with status unknown. 
[YYYY-MM-DDTHH:MM:SS] debug wcp [kubelifecycle/node_reflector.go:124] Handling update for node 4200##########################20 with status unknown.

Environment

VMware vSphere Kubernetes Service

Cause

This issue occurs when a non-critical baseline attaches to the Supervisor Cluster, resulting in the installation of an incompatible spherelet version on the control plane nodes.

Resolution

Detach the offending baseline to address the incompatible spherelet version and ensuring the Supervisor version remains current.

1. Detach the non-critical baseline
   

   Remove the predefined host patch baselines from the cluster to prevent further installation of incompatible spherelet components.

   Review and detach the following baselines:

   • Non-Critical Host Patches (Predefined)

   • Host Security Patches (Predefined)

   • Critical Host Patches (Predefined)

2. Upgrade Supervisor

Upgrade the Supervisor Cluster to a compatible version to ensure the correct spherelet versions deploy across the environment. For detailed instructions, see Upgrade a Supervisor.