vCenter Server becomes unstable and services fail to start with "no space left on device" error despite the df command showing sufficient free space.

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• After rebooting the vCenter Server, some services fail to start.
• File creation fails with a "No space left on device" error, even though the df command indicates that sufficient free space is available.
  Example:

• root@vcsa [ ~ ]# df .
  Filesystem                      1K-blocks     Used Available Use% Mounted on
  /dev/mapper/vg_root_0-lv_root_0  49222292 33944412  12745120  73% /
  root@vcsa [ ~ ]# echo test > test.txt
  bash: echo: write error: No space left on device

• The mount point /storage/lvm_snapshot is not present in the df command output.
• When executing the lvs command, LVM Snapshots are present, and some show a Snap% (snapshot space usage) that has reached 100%.
  Example:

  root@vcsa [ ~ ]# lvs -o lv_name ,vg_name,lsv_size,snap_percent,origin
    LV                                VG              LSize    Snap%  Origin      
    archive                           archive_vg        49.99g                    
    archive_vg_archive_snap           archive_vg         4.71g 0.01   archive     
  ...
    lv_root_0                         vg_root_0        <48.00g                    
    vg_root_0_lv_root_0_snap          vg_root_0        <10.30g 100.00 lv_root_0
  ...

  The presence of LVs named with the *_snap suffix indicates that LVM Snapshots exist on the vCenter Server. In the example above, the Snap% for vg_root_0_lv_root_0_snap has reached 100%, which prevents any new data from being written to the root file system.

Environment

VMware vCenter Server 7.0 U3

VMware vCenter Server 8.0

Cause

This issue occurs when an LVM snapshot-based auto-backup is created during vCenter Server patching but is not deleted correctly. These snapshots are intended as rollback points and should be removed automatically after the update. However, if they remain, the snapshot space will eventually reach 100%, which prevents any new data from being written to the file system.

Resolution

Depending on the impacted file systems, the vCenter Server may be unrecoverable and you may need to restore the vCenter Server from backup. To attempt a manual recovery, follow these steps:

1. Create an offline snapshot of the vCenter Server. Ensure you follow the guidelines in Snapshot Best practices for vCenter Server Virtual Machines.
2. Remove the LVM snapshots using the script attached to the following KB article: "LVM Snapshot based auto-backup of vCenter Server will not be taken during vcenter patching", pre-check warning message during vCenter Server patching
3. Once the LVM snapshots are deleted, confirm that all vCenter Server services start correctly and the Management Interface (VAMI) or vSphere Client is accessible.

Note: If the vCenter Server remains unstable or services fail to start after the LVM snapshots are removed, a restore from a file-based backup is required. If you require further assistance, contact Broadcom Technical Support.

Additional Information

vCenter Server が不安定になり、df コマンドの結果で十分な空き容量があるにもかかわらず "no space left on device" というエラーでサービスの起動に失敗する