A worker node of a VKS workload cluster is in a read-only state

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Application and system pods of a VKS workload cluster may be seen with a consistent Status of ContainerCreating or CrashLoopBackOff.

 

Describing a pod in a CrashLoopBackOff status displays a kubelet error similar to the following:

• Error: open /var/lib/kubelet/pods/########-####-####-####-############/etc-hosts: read-only file system

 

Viewing the console of the affected VKS workload cluster node may list errors similar to the following:

• EXT4-fs (sdc#):  Remounting filesystem read-only

Environment

• vSphere Supervisor 8.x

Cause

• Power failures or storage failures within a datacenter may cause VKS nodes to enter a read-only state.
• Root disk usage on a VKS node reaching 100% may also cause the node to enter a read-only state.

Resolution

Power cycle the affected VKS workload cluster node using the vSphere Client or ESXi Host Client.

If the affected VKS workload cluster node remains in a read-only state after being power cycled, please open a support case with Broadcom Technical Support to further troubleshoot the issue.