Pod in terminating status after storage outage

Products

• VMware Telco Cloud Automation

Issue/Introduction

• Pod not coming up
• Pod stuck in terminating status
• Environment had a storage outage
• Kubelet logs show message similar to:
  

  EnvFrom: [)EnvFromSource(),TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice(},StartupProbe:nil, ) start failed in pod <pod name>: CreateContainerConfigError: open /var/lib/kubelet/pods/########-####-####-####-############/etc-hosts: read-only file system

kubelet[974]: E0731 15:36:20.237060 974 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err-"orphaned pod \" ########-####-####-####-############\" found, but error occurred when trying to remove the volumes dir: read-only file system" numErrs-1

Environment

2.3.x

Cause

Due to the storage outage the node that the pod was on had a read-only filesystem.

Resolution

Reboot the node that is experiencing the Read-Only Filesystem so the OS can remount the filesystem Read/Write.