The storage/core filesystem is out of disk space or inodes alert in vCenter Server Appliance Health Status

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• The storage/core filesystem is out of disk space or inodes
• Logging into the VCSA via SSH and running ‘df –h’ shows that  /storage/core is 100% utilized
• Performing the command "ls -lah" on the /storage/core directory shows log bundles (filenames and timestamps may vary and are included for example purposes only):

vcsa01:/storage/core # df -h
Filesystem                            Size  Used Avail Use% Mounted on
/dev/sda3                              11G  3.9G  6.3G  39% /
udev                                  4.0G  168K  4.0G   1% /dev
tmpfs                                 4.0G   40K  4.0G   1% /dev/shm
/dev/sda1                             128M   38M   84M  31% /boot
/dev/mapper/core_vg-core               25G   25G    0M 100% /storage/core
/dev/mapper/log_vg-log                9.9G  209M  9.2G   3% /storage/log
/dev/mapper/db_vg-db                  9.9G  192M  9.2G   3% /storage/db
/dev/mapper/dblog_vg-dblog            5.0G  171M  4.5G   4% /storage/dblog
/dev/mapper/seat_vg-seat              9.9G  165M  9.2G   2% /storage/seat
/dev/mapper/netdump_vg-netdump       1001M   18M  932M   2% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy  9.9G  151M  9.2G   2% /storage/autodeploy
/dev/mapper/invsvc_vg-invsvc          5.0G  146M  4.6G   4% /storage/invsvc

Environment

VMware vCenter Server Appliance 6.0.x

Cause

Log bundles created by vm-support command and/or by "export system logs" in the vSphere Client are not removed when completed.

Resolution

1. Take a snapshot of the vCenter Server Appliance
2. Stop the vpxd service by running the command service vmware-vpxd stop
3. Verify the service has stopped by running the command service vmware-vpxd status; you should see "vmware-vpxd is stopped"
4. Remove the .tgz files by using the ‘rm’ command. For example: rm vc-vcsa10-216-08-28-14.17.tgz
5. Repeat as needed.
6. Once this is done, confirm that there is now free space in the /storage/core area by running the "df -h" command:

Filesystem                            Size  Used Avail Use% Mounted on /dev/sda3                             
 /udev                                  4.0G  168K  4.0G   1% /dev
/tmpfs                                 4.0G   40K  4.0G   1% /dev/shm
/dev/sda1                             128M   38M   84M  31% /boot
/dev/mapper/core_vg-core               25G  173M   24G   1% /storage/core
/dev/mapper/log_vg-log                9.9G  209M  9.2G   3% /storage/log/dev/mapper/db_vg-db                  9.9G  192M  9.2G   3% /storage/db
/dev/mapper/dblog_vg-dblog            5.0G  171M  4.5G   4% /storage/dblog
/dev/mapper/seat_vg-seat              9.9G  165M  9.2G   2% /storage/seat
/dev/mapper/netdump_vg-netdump       1001M   18M  932M   2% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy  9.9G  151M  9.2G   2% /storage/autodeploy
/dev/mapper/invsvc_vg-invsvc          5.0G  146M  4.6G   4% /storage/invsvc
 

1. Start the vpxd service by running the command “service vmware-vpxd start.”  The error alarm should now clear in vCenter. 

Additional Information

Impact/Risks:
This process can cause irreversible damage to files if done incorrectly.If in doubt, open a Support Request with VMware Technical Support!

Can fill up the available space on the vCSA causing the web client to become unresponsive and/or not display the inventory.