VCO Disk Expansion Issue - 'resize2fs: Permission denied to resize filesystem'

Products

• VMWare SD-WAN
• VMware SD-WAN by VeloCloud
• VMware VeloCloud SD-WAN

Issue/Introduction

Symptoms:

Customers increase the disk space on their VeloCloud Orchestrator by following https://docs.vmware.com/en/VMware-SD-WAN/4.5/sd-wan-orchestrator-deployment-and-monitoring-guide/GUID-A1ED5E3C-2B5D-4719-9D5B-143BB3F47FE2.html.

In step 10,

When attempting to extend a logical volume using the lvextend command, the following error is encountered:

resize2fs: Permission denied to resize filesystem

Environment

VMware SD-WAN Orchestrator (VECO)

Cause

The error occurs because:

1. The file system (EXT4) has internal block-level errors.

2. These errors must be corrected before resizing operations can proceed.

3. Online resizing is not allowed when there are unresolved errors in the file system.

Error messages in /var/log/messages and dmesg output will be like below :-

EXT4-fs (xvdh): error count since last fsck: 1
EXT4-fs (xvdh): initial error at time 1587778646: ext4_free_blocks:4885
EXT4-fs (xvdh): last error at time 1587778646: ext4_free_blocks:4885
[43567001.312562] EXT4-fs warning (device xvdh): ext4_resize_begin:44: There are errors in the filesystem, so online resizing is not allowed 

 

Resolution

Follow these steps to resolve the issue:

Step 1: Identify and Unmount the Affected File System

1. Unmount the file system:

   umount -l /dev/mapper/store3-data

2. Identify processes running on the file system:

   fuser -v -m /dev/mapper/store3-data

3. Kill or stop the services using the file system. For example, if clickhouse is running:

   service clickhouse-server stop

Step 2: Perform File System Check and Repair

1. Run the file system check and repair:

   e2fsck -y /dev/mapper/store3-data

   • Note: This process can take a significant amount of time depending on the size of the file system.

Step 3: Resize the File System

1. Resize the file system:

   resize2fs /dev/mapper/store3-data

Step 4: Mount the File System Back

1. Mount the file system:

   mount /dev/mapper/store3-data /store3

2. Verify the resized file system:

   df -h

Additional Information

Risks:

1. Data Loss: File system errors or improper handling during repair can result in data loss.

   • Mitigation: Always back up critical data before starting the repair process.

2. Extended Downtime: Repairing and resizing large file systems can take considerable time.

   • Mitigation: Schedule the operation during a maintenance window to minimize impact on production.

3. Service Disruption: Stopping services to unmount the file system may disrupt dependent applications.

   • Mitigation: Plan service outages in advance.

4. Incomplete Repairs: Failure to resolve all block-level errors might cause issues during resizing.

   • Mitigation: Double-check the e2fsck output and rerun the command if necessary until no errors remain.

 

Additional Notes

• Ensure that all services dependent on the affected file system are stopped before proceeding.

• It is recommended to perform a backup of the file system before initiating any repair or resize operations.

• The e2fsck command requires unmounted file systems to prevent data corruption.