etcd and kube-apiserver pods are in CrashLoopBackOff on Guest Cluster after a power outage event

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• etcd and kube-apiserver pods are in CrashLoopBackOff in Guest cluster.
• This is observed after a power outage. This can also happen after a node power off.
• etcd container keep restarting. kube-apiserver container also keep restarting.
• etcd container logs shows below error

YYYY-MM-DDTHH:MM:SS.092110285Z stderr F panic: assertion failed: Page expected to be: <#123#>, but self identifies as <#1234567#>

• • To retrieve container logs:

crictl logs <container id>

Environment

vSphere with Tanzu

VKS

VKS Guest Cluster

Cause

This error suggests that a specific page within the database file is expected to have a certain identifier or state, but it self-identifies differently, leading to an assertion failure and a program panic.

Resolution

Reference Troubleshooting ETCD Issues in vSphere Supervisor / vSphere Kubernetes Service (VKS)