Expanding a persistent volume fails when the file system on the volume is corrupted

Products

• VMware vSphere with Tanzu
• vSphere with Tanzu
• VMware vCenter Server

Issue/Introduction

Expanding a volume fails and the pod is unable to mount the volume.

Symptoms:

• The file system on the volume has not been resized after the volume size has been expanded.
• Pod remains in the pending state

Describing the pod shows the following error:

Warning  FailedMount  3m33s (x38 over 78m)  kubelet  MountVolume.MountDevice failed while expanding volume for volume "pvc-a45507f6-f70c-42b1-87d4-###########" :
Expander.NodeExpand failed to expand the volume : rpc error: code = Internal desc = error when resizing filesystem on volume "d106b588-b94e-4548-ad89-###########-a45507f6-f70c-42b1-87d4-############" on node: error when resizing filesystem on devicePath /dev/sdf and volumePath /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a45507f6-f70c-42b1-87d4-############/globalmount, err: resize of device /dev/sdf failed: exit status 1. resize2fs output: resize2fs 1.45.6 (20-Mar-2020)
resize2fs: Permission denied to resize filesystem
Filesystem at /dev/sdf is mounted on /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a45507f6-f70c-42b1-87d4-############/globalmount; on-line resizing required
old_desc_blocks = 3, new_desc_blocks = 7

Environment

vSphere with Tanzu

Cloud Native Storage

VMware vCenter Server

Cause

Running dumpe2fs on the volume reports the following filesystem state:

dumpe2fs /dev/sdf | grep -i error
dumpe2fs 1.45.5 (07-Jan-2020)
Filesystem state:         clean with errors
Errors behavior:          Continue
FS Error count:           3

A Filesystem state "clean with errors" indicates that filesystem superblock is aware of potential problems on the filesystem.
The recommended action in this situation is to run fsck as soon as possible.

Although it may be possible to mount the filesystem in this state, it is strongly discouraged to mount a  filesystem with errors in order to prevent data corruption.

Resolution

Fix filesystem errors for the filesystem resize to complete and for the volume mount to succeed.

1. From the TKC K8s Cluster, look up VolumeAttachment objects and see which TKC node the PV is attached to
   
    # kubectl get volumeattachment | grep pvc-a45507f6-f70c-42b1-87d4-############

2.  

   SSH into TKC node VM and identify the mount point on the TKC node for the PV
   
   # mount | grep pvc-a45507f6-f70c-42b1-87d4-############
/dev/sdf on /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a45507f6-f70c-42b1-87d4-############/globalmount type ext4 (rw,relatime)

3.  

   From the same TKC node SSH terminal, execute following command to Identify device for PVC
   
   # lsblk | grep pvc-a45507f6-f70c-42b1-87d4-7c36bdbb0c3e
sdf    8:128  0   50G  0 disk /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a45507f6-f70c-42b1-87d4-###########/globalmount

4.  

   Unmount the volume
   
   # umount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a45507f6-f70c-42b1-87d4-############/globalmount

5.  

   Check if file system has errors using the device identified in step-3
   
   # dumpe2fs /dev/sdf | grep -i error

6. Run filesystem check on the device identified in step-3
   
   # e2fsck -y /dev/sdf

7. Delete the pending Pod and wait for it be recreated in the running state.