Error "MountVolume.SetUp failed for volume "<#########>" : mkdir /var/lib/kubelet/pods/#########-####-####-####-############: read-only file system" during pod failure

Products

• VMware Telco Cloud Automation
• VMware Tanzu Kubernetes Grid
• VMware Telco Cloud Platform

Issue/Introduction

• Pods in the Tanzu Kubernetes Grid workload cluster remain stuck in Init or Terminating states for an extended period.
  
  
• When inspecting the affected pods using kubectl describe pod, the kubelet reports the following FailedMount and Failed events.
• Container logs for components such as upgrade-ipam and install-cni indicate they are waiting to start: PodInitializing.

Environment

TCA 3.4
TKGm 2.5.4
TCP 5.1

Cause

The underlying worker node operating system has entered a read-only filesystem state, causing stale pods from previous reboots to become stuck in a continuous loop attempting to allocate data directories.

Resolution

1. Identify and force delete any stuck pods across all namespaces to clear existing unrecoverable objects. First, list the affected pods:
   
   kubectl get pods -A | grep -E "Init|Terminating|ContainerCreating"
   
   
2. Force delete the identified stuck pods:
   
   kubectl delete pod <POD_NAME> -n <NAMESPACE> --force --grace-period=0
   
   
3. Identify the specific worker nodes exhibiting the read-only file system behavior by checking cluster events and pod placements:
   
   kubectl get events -A | grep "read-only file system" | awk '{print $4}' | sort | uniq

kubectl get pods -A -o wide | awk '/Init|Terminating|ContainerCreating/ {print $8}' | sort | uniq
   
   
4. Log into the vCenter Server and perform a guest OS reboot of the worker node VMs identified in Step 3.
   
   
5. Verify pod recovery post-reboot by ensuring pods across the cluster successfully initialize and no pods remain stuck:
   
   kubectl get pods -A | grep -E "Init|Terminating|ContainerCreating"