Persistent volume mounted on a Windows worker node fails to attach to a Linux worker node with error "failed to mount volume as "ext4"; already contains unknown data, probably partitions: error: mount failed: exit status 32"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• The pods (irrespective of being deployment or stateful-set) are stuck in "Container Creating" state.
  
  
• Per the describe output of the affected pod, the CSI controller's attach-detach controller is able to attach the volume to the node. However, the kubelet inside the node is unable to format and mount it because of the unknown partition and data inside it.
  
    Normal   SuccessfulAttachVolume  <time in sec>        attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-<ID>"
  Warning  FailedMount             <time in sec>        kubelet                  MountVolume.MountDevice failed for volume "pvc-<ID>" : rpc error: code = Internal desc = error in formating and mounting volume. Parameters: {<volume-handle ID> ext4 /var/lib/kubelet/plugins/kubernetes.io/csi/csi.vsphere.vmware.com/<ID>/globalmount [] false} err: failed to mount volume as "ext4"; already contains unknown data, probably partitions: error: mount failed: exit status 32
mounting arguments: -t ext4 -o defaults /dev/disk/by-id/wwn-<ID> /var/lib/kubelet/plugins/kubernetes.io/csi/csi.vsphere.vmware.com/<ID>/global mount
output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/csi.vsphere.vmware.com/<ID>/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error.
  
  

Environment

VMware vSphere Kubernetes Service
VMware vSphere Container Storage Plugin

Cause

The persistent volumes are initially formatted as Windows Filesystem (probably NTFS) considering they were attached and mounted on Windows worker nodes. The CSI driver attaches the disk to the node however the Linux kernel identifies the filesystem mismatch and therefore rejects the request to mount it as ext4 since it may lead to data corruption or any other possible failures. 

As a result, the volume mount within the node fails and no globalmount point is created which can then be attached to the pod for the container to start successfully.

Resolution

1. It is highly recommended to create separate volumes for Linux and windows worker nodes.
   
   
2. Do NOT reformat the existing volume since it can result in data loss and make the volume irrecoverable.