vCenter Server Appliance disk space is full
search cancel

vCenter Server Appliance disk space is full

book

Article ID: 318953

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

This article provides information to troubleshoot vCenter Server Appliance (VCSA) high virtual disk space.

  • Errors appear as messages in vCenter Server similar to:

    • vSphere UI Health Alarm", "Log disk exhaustion on vcenter name"

    • Database Health Alarm", "Core and Inventory Disk Exhaustion on vcenter name"

  • By default, the error messages begin when the disk reaches 80% full

    • Please note that in version 6.7 and later versions, the /storage/archive disk can be up to 100% full by design

    • Earlier 6.7 versions may still present an error in the vCenter GUI. These can be safely ignored.

  • By default, when the some disks reach 95% full, the main vCenter service, vmware-vpxd, is turned off automatically

    • This is to protect the database from corruption.

    • Thus the problem may cause vCenter Server to be inaccessible with 503 Service Unavailable errors.

There are many possible causes for specific disk partitions running out of space. Some causes include:

  • Partitions are not sized correctly

  • Large files in the partitions

  • Tables in the database consuming space

  • Services are failing to clean up files

Environment

VMware vCenter Server

Resolution

Identify the full partition

 

  • Through the vCenter GUI:

  1. Log In to the vCenter Server Appliance Management Interface (VAMI)​​​​​​

  2. Select Monitor > Disks

  3. Note which disks are showing 78% or higher usage

  4. Please keep in mind that in 6.7 and higher versions, the "archive" disk being 80% or more full can be safely ignored​​​​

  • Through the vCenter command line:

  1. Log into the vCenter Server Appliance as root via SSH or the vCenter virtual machine Console.

    Note: If unable to log in to vCenter Appliance and suspect this is due to the /dev/sda3 "root partition" being full, refer to the following article steps 1 to 7 to gain Single user mode shell access:  Resetting root password in vCenter Server Appliance 6.5 / 6.7 / 7.x / 8.x.
     

  2. Type shell to switch to the BASH Shell. (for more information about enable or disable SSH ad Bash shell access, refer Enable or Disable SSH and Bash Shell Access

  3. Find out which partitions are full enough to trigger warnings. 78% or more full by running the command:

df -h

Here's an alternative command which only shows the partitions which are getting full:

df -h |awk '0+$5 >= 78 {print}'

Note: Searching for "78" because in many cases if the partition is at 78-79%, it may be exceeding the 80% needed to trigger warnings, then recovering back to the lower percentage. Similarly, if it is at 93%, it may be exceeding the 95% needed to trigger vmware-vpxd service to shut down, then recovering back to the lower percentage.

 

 

Further troubleshooting:

  1. If the previous links did not help with the problem, Identify large files in the partition

High disk space usage in vCenter Server Appliance starts with identifying which partition is impacted with high disk space usage. 

  1. Connect to vCenter Server Appliance through SSH or through the vCenter VM console

  2. Type shell to switch to the BASH Shell. (for more information about enable or disable SSH ad Bash shell access, refer Enable or Disable SSH and Bash Shell Access

  3. Use df -h command to get the disk space usage stats of the vCenter Server Appliance 

Note: If unable to access VCSA bash shell, refer Unable to log in to the vCenter Server Appliance shell using root account even after password reset

 

Once the partition with low disk space is identified, below steps can be followed to identify the large files created in the respective partitions, except for the partitions store database files. 

 

For partitions and mount points of vCenter Server Appliance, refer: 

  1.  Use the below command within the affected partition to identify top 20 files with high disk usage: 

du -a <Path> | sort -n -r | head -n 20 
 

For example: 

du -a /storage/log/ | sort -n -r | head -n 20

  1. Change directories to any listed here which are suspiciously large, and look for large files using the command:

ls -lha

There is also a more powerful search using a more complex command for finding files larger than 100MB:
find ./ -type f -size +100M |less
 
If large files are not the issue it may be that numerous smaller files are causing a problem. The following command will display the subdirectories containing the most files:

find ./ -type d -exec sh -c 'echo -n "{}: " && find "{}" -type f | wc -l' \; | awk '$2 > 100' | sort -k2,2nr |less

Please note, this command can take a few minutes to run!

Before deciding to remove any files to clear space, ensure there is a reliable backup of vCenter.

Note: The *.tgz files are archived log bundle files and can be removed.

 

Resizing the vCenter virtual disks

If all else fails, it can be that normal activity on the vCenter has become greater than the existing disk size supports., thus requiring to resize the virtual disk:

 

 

Additional Information

Powered by Wolken Software