APIServerPodHealthy/EtcdPodHealthy condition is false when upgrading vSphere Kubernetes Release

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Upgrading vkr version failed in preflight checks.

Waiting for control plane to pass preflight checks to continue reconciliation: [machine xxxx-control-plane-xxxxx reports APIServerPod Healthy condition is false (Error: CrashLoopBackOff), machine xxxx-control-plane-xxxxx reports EtcdMemberHealthy condition is false (Error: CrashLoopBackOff), machine xxxx-control-plane-xxxxx reports EtcdMemberHealthy condition is unknown (Failed to connect to the etcd pod on the xxxx-control-plane-xxxxx node: could not establish a connection to any etcd node: context deadline exceeded)]

etcd and apiserver pod in this control plane node in CrashLoopBackOff status.

etcd logs shows unable to get etcd cluster response with a non-exist IP address.

{"level":"warn","ts":"","caller":"etcdserver/cluster_util.go:79","msg":"failed to get cluster response","address":"https://<non-exist-IP>:2380/members","error":"Get \"https://non-exist-IP:2380/members\": dial tcp non-exist-IP:2380: i/o timeout"}

Environment

VMware vSphere Kubernetes Service

Cause

Unexpected shutdown/removal of control plane node causing etcd not updating cluster member.

Resolution

Reboot this control plane node.

Verify etcd/apiserver is running post reboot.