Error: "reading directory '.': bad message" when trying to access a directory in a PVC attached to a container

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• When accessing a directory (either ls or cd) on a PVC attached to a container, the following error is present within the container:
  
  reading directory '.': bad message
  
  
• Other directories within the PVC are accessible

Cause

This is due to some file level corruption on the PVC object.

Resolution

1. Restore the PVC from a good known backup, confirming the filesystem can access the specific directory after the restore.
   
   
2.  Locate the VMDK backing the PVC, then use an linux rescue CD on a VM to mount the backing VMDK and try to use fsck on the filesystem for the VMDK. Even though this is a PVC from a VKS / Kubernetes perspective, the underlying filesystem is EXT4 and can be mounted and repaired. 

If many PVC's are found the have intermittent corrupt directories, this is indicative of back end infrastructure issues and should be investigated accordingly.

Additional Information

Monitor Container Volumes in vSAN Cluster