vCenter Server Appliance disk space is full

Products

• VMware vCenter Server

Issue/Introduction

This article provides information to troubleshoot vCenter Server Appliance (VCSA) high virtual disk space.

• Errors appear as messages in vCenter Server similar to:

  • vSphere UI Health Alarm", "Log disk exhaustion on vcenter name"

  • Database Health Alarm", "Core and Inventory Disk Exhaustion on vcenter name"

• By default, the error messages begin when the disk reaches 80% full

  • Please note that in version 6.7 and later versions, the /storage/archive disk can be up to 100% full by design

• By default, when the some disks reach 95% full, the main vCenter service, vmware-vpxd, is turned off automatically

  • This is to protect the database from corruption.

  • Thus the problem may cause vCenter Server to be inaccessible with 503 Service Unavailable errors.

There are many possible causes for specific disk partitions running out of space. Some causes include:

• Partitions are not sized correctly

• Large files in the partitions

• Tables in the database consuming space

• Services are failing to clean up files

Environment

VMware vCenter Server

Resolution

Identify the full partition

• Through the vCenter GUI:

1. Log In to the vCenter Server Appliance Management Interface (VAMI)​​​​​​

2. Select Monitor > Disks

3. Note which disks are showing 78% or higher usage. 
   The "archive" disk usage over 80% is by design and can be safely ignored​​​​

• Through the vCenter command line:

1. Log into the vCenter Server Appliance as root via SSH or the vCenter virtual machine Console.
   
   Note: If unable to log in to vCenter Appliance and suspect this is due to the /dev/sda3 "root partition" being full, refer to the following article steps 1 to 7 to gain Single user mode shell access:  Resetting root password in vCenter Server Appliance 6.5 / 6.7 / 7.x / 8.x (322247).

2. Type shell to switch to the BASH Shell. (for more information about enable or disable SSH ad Bash shell access, refer Enable or Disable SSH and Bash Shell Access

3. Find out which partitions are full enough to trigger warnings. 78% or more full by running the command:

df -h

Here's an alternative command which only shows the partitions which are getting full:

df -h |awk '0+$5 >= 78 {print}'

Note: Searching for "78" because in many cases if the partition is at 78-79%, it may be exceeding the 80% needed to trigger warnings, then recovering back to the lower percentage. Similarly, if it is at 93%, it may be exceeding the 95% needed to trigger vmware-vpxd service to shut down, then recovering back to the lower percentage.

• Select the article links for partitions that are displaying as full from this list

  Filesystem	Mounted on	Resolution
  /dev/sda3	"/"	/dev/sda3 - root partition 100% full due to Audit.log files not being rotated in vCenter Server Appliance (318911)   Root Partition full due to dnsmasq.log files in vCenter Server Appliance or Platform Service Controller (341157)
  /dev/mapper/log_vg-log	/storage/log	Troubleshooting vCenter Appliance /storage/log directory is 80% or more full (313077)
  /dev/mapper/core_vg-core	/storage/core	/storage/core partition full due to core.netlogond.xxxxx files in vCenter Server Appliance (318750)
  /dev/mapper/db_vg-db	/storage/db	/storage/db full or nearly full in VCSA (318880)
  /dev/mapper/seat_vg-seat	/storage/seat	/storage/seat disk 100% full on vCenter Server Appliance 6.x/7.x (318931)
  /dev/mapper/archive_vg-archive	/storage/archive	Note: This partition being full is normal and expected in vCenter versions 6.7 and above "File system /storage/archive is low on storage space" warning in vCenter Server Appliance (318803)

Further troubleshooting:

1. If the previous links did not help with the problem, Identify large files in the partition

High disk space usage in vCenter Server Appliance starts with identifying which partition is impacted with high disk space usage. 

1. Connect to vCenter Server Appliance through SSH or through the vCenter VM console

2. Type shell to switch to the BASH Shell. (for more information about enable or disable SSH ad Bash shell access, refer Enable or Disable SSH and Bash Shell Access

3. Use df -h command to get the disk space usage stats of the vCenter Server Appliance 

Note: If unable to access VCSA bash shell, refer Unable to log in to the vCenter Server Appliance shell using root account even after password reset (343642)

Once the partition with low disk space is identified, below steps can be followed to identify the large files created in the respective partitions, except for the partitions store database files. 

For partitions and mount points of vCenter Server Appliance, refer: 

• List of VMDKs/Partitions for a vCenter Server Appliance 7.0 - Size Mount point and Purpose (322801)

• List of VMDKs/Partitions for a vCenter Server 8.0 - Size Mount point and Purpose (316438)

2.  Use the below command within the affected partition to identify top 20 files with high disk usage: 

du -a <Path> | sort -n -r | head -n 20 

For example: 

du -a /storage/log/ | sort -n -r | head -n 20

3. Change directories to any listed here which are suspiciously large, and look for large files using the command:

ls -lha

There is also a more powerful search using a more complex command for finding files larger than 100MB:
find ./ -type f -size +100M |less
 
If large files are not the issue it may be that numerous smaller files are causing a problem. The following command will display the subdirectories containing the most files:

find ./ -type d -exec sh -c 'echo -n "{}: " && find "{}" -type f | wc -l' \; | awk '$2 > 100' | sort -k2,2nr |less

Please note, this command can take a few minutes to run!

Before deciding to remove any files to clear space, ensure there is a reliable backup of vCenter.

Note: The *.tgz files are archived log bundle files and can be removed.

Resizing the vCenter virtual disks

If all else fails, it can be that normal activity on the vCenter has become greater than the existing disk size supports., thus requiring to resize the virtual disk:

• Notice that in the VAMI the identity of the exact virtual disk to enlarge is shown in Monitor > Disks

  • Otherwise, please refer to the "List of VMDKs/Partitions" articles linked above 

• Please keep in mind the disk size on the vCenter VM cannot be changed if there is a snapshot of the VM

  • It is critical to ensure a good backup of vCenter has been made before resizing any virtual disks on it

• Increasing the disk space for the VMware vCenter Server Appliance in vSphere 6.5 and above (316602)

Additional Information

Monitor and prevent vCenter Appliance disks from filling (318930)