Error "Back-off restarting failed container kube-apiserver in pod <node name>" observed when kube-apiserver pod is stuck in CrashLoopBackoff state

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

• The Kube APIServer pod is cycling in a CrashLoopBackOff state.
  
  
• kubectl describe <kube-apiserver pod name> -n name space shows the below error
  
  Error "Back-off restarting failed container kube-apiserver in pod <node name>" 
  
  
• The Control Plane node count is degraded (e.g., 2 nodes instead of the configured 3).
  
  
• Scaling operations (up or down) fail to stabilize the cluster
  
  
  • kubectl describe machine <machine-name> -n <namespace> has the below error 
    
    

    Conditions:
        Last Transition Time:  2025-12-15T23:37:41Z
        Status:                True
        Type:                  Ready
        Last Transition Time:  2025-12-19T05:22:57Z
        Message:               CrashLoopBackOff
        Reason:                PodFailed
        Severity:              Error
        Status:                False
        Type:                  APIServerPodHealthy
        Last Transition Time:  2025-12-15T23:12:04Z
        Status:                True
        Type:                  BootstrapReady
        Last Transition Time:  2025-12-19T05:09:30Z
        Status:                True
        Type:                  ControllerManagerPodHealthy
        Last Transition Time:  2025-12-19T05:18:05Z
        Message:               Missing etcd member
        Reason:                EtcdMemberUnhealthy
        Severity:              Error
        Status:                False
        Type:                  EtcdMemberHealthy
        Last Transition Time:  2025-12-19T05:09:30Z
        Message:               Pod etcd-######### is missing
        Reason:                PodMissing
        Severity:              Error
        Status:                False
        Type:                  EtcdPodHealthy
    

• etcd.yaml file is missing in ls -l "/etc/kubernetes/manifests/" in the worker node

Environment

2.x

Cause

• The Kubelet daemon relies on static manifests in /etc/kubernetes/manifests/ to start critical control plane components.
• Without etcd.yaml, the Kubelet cannot start the etcd member on that node.
• Since the kube-apiserver depends on a healthy and reachable local etcd member (or quorum), it fails to initialize and enters a crash loop.

Resolution

Delete the machine set and this will trigger a new machine replacement with the etcd.yaml manifest file.

1. Delete the machine identified from the error message

kubectl delete machine <machine-name> -n <namespace>
   
   
2. Check if the machine is recreated
   
   kubectl get machines -n <namespace> -w