VCO Disk Expansion Issue - 'resize2fs: Permission denied to resize filesystem'

Products

• VMware

Issue/Introduction

Symptoms:

Customers increase the disk space on their VeloCloud Orchestrator by following https://docs.vmware.com/en/VMware-SD-WAN/4.5/sd-wan-orchestrator-deployment-and-monitoring-guide/GUID-A1ED5E3C-2B5D-4719-9D5B-143BB3F47FE2.html.

In step 10, after running lvextend command, they may get errors like 'resize2fs: Permission denied to resize filesystem'.

 

Environment

VMware SD-WAN

Cause

It usually happens when there are internal block level errors.

 

Error messages in /var/log/messages and dmesg output will be like below :-

 

 

EXT4-fs (xvdh): error count since last fsck: 1
EXT4-fs (xvdh): initial error at time 1587778646: ext4_free_blocks:4885
EXT4-fs (xvdh): last error at time 1587778646: ext4_free_blocks:4885

[43567001.312562] EXT4-fs warning (device xvdh): ext4_resize_begin:44: There are errors in the filesystem, so online resizing is not allowed

 

Resolution

Since file system has internal block level errors we need to un-mount the file system then run the file system check and correct block level errors.

# unmount -l /dev/mapper/store3-data

We need to run below to check process which is running on /dev/mapper/store3-data. We need to kill or shutdown the services running on this filesystem otherwise next step of file system check won't work.

# fuser -v -m /dev/mapper/store3-data

In my case only clickhouse service was running so stopped the clickhouse service (service clickhouse-server stop) and then proceeded with below.

# e2fsck -y /dev/mapper/store3-data

Above command will take longer time based on the FS size.

 

 

Resize the file system before mounting it back

# resize2fs /dev/mapper/store3-data

After successful resize, I mounted it back

# mount /dev/mapper/store3-data /store3

Now we can run 'df -h' and check that the file system is expanded successfully

 

Additional Information

Impact/Risks:

File system cannot be expanded to the error. If the filesystem is a critical file system, it will cause production down.