New Pods Stuck in "Pod Initializing" due to NFS Volume Mount Timeout and Containerd Deadlock

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

• New pods fail to start and hang indefinitely in the Init state.

• kubectl describe pod shows State: Waiting and Reason: PodInitializing with no events generated.

• Existing, running pods on the same worker nodes are completely unaffected and continue operating normally.

• Running kubectl logs <pod-name> -c <init-container-name> returns an API error:  'Error from server (BadRequest): container "<name>" in pod "<name>" is waiting to start: PodInitializing.'

• Kubelet Logs on the worker nodes shows
  
  Error Syncing pod. skipping' err="unmounted volumes=[logs], unattched volumes = [], failed to process volumes = [], context deadline exceeded"

• kubectl get pvc -A  command shows that all the PVC are in bound state and no errors seen

Environment

TKGm 2.5.2

Cause

Kubelet's attempt to reach the NFS server to mount the pod's persistent volume hung indefinitely over the network. This unresolvable network I/O block led to kubelet timeouts (context deadline exceeded)
As a result, new pods were completely blocked from starting and hanging in Pod Initializing state while existing pods are continue working healthy since it already had established network and storage connections prior to the network disruption. 
The root cause is a network connectivity failure between the TKGM worker node and the underlying NFS server.

Resolution

Engage Network Team and ensure proper IP routing and firewall rules allowed between the TKGM worker nodes and the NFS storage appliance.

Additional Information

User might notice "Memory cgroup out of memory: killed process" on the VM console of TKGm worker node from vCenter UI though the Memory Utilization doesn't hit the maximum.