/Storage/Core filling up - vg_root_0-lv_root_0 causing services on vCenter to fail

Products

• VMware vCenter Server

Issue/Introduction

Storage core has reached over 80% usage. We find that it is all being reported in vg_root_0-lv_root_0.

When running the df -h command we see the following:

root@vcentername [ ~ ]# df -h
Filesystem                                Size  Used Avail Use% Mounted on
devtmpfs                                  9.3G     0  9.3G   0% /dev
tmpfs                                     9.3G  4.0K  9.3G   1% /dev/shm
tmpfs                                     9.3G  1.2M  9.3G   1% /run
tmpfs                                     9.3G     0  9.3G   0% /sys/fs/cgroup
/dev/mapper/vg_root_0-lv_root_0            47G   45G     0 100% /
tmpfs                                     9.3G     0  9.3G   0% /tmp

Listing the 10 largest files in the partition, which gives us an indication of the culprit:
root@vcentername [ / ]# find . -type f -print0 | xargs -0 du -h | sort -rh | head -n 10
...data omitted...
37G     ./var/vmware/applmgmt/appliance_stats.sqlite-wal
...data omitted...

The appliance_stats.sqlite files in a VMware vCenter Server Appliance (VCSA) is a SQLite database file used by the statsmonitor service to store various statistics and performance-related data for the vCenter Server Appliance itself, and its integrity is checked during service initialization. This database plays a role in the overall health and monitoring of the VCSA and its supporting services. These files are:

• /var/vmware/applmgmt/appliance_stats.sqlite
• /var/vmware/applmgmt/appliance_stats.sqlite-shm
• /var/vmware/applmgmt/appliance_stats.sqlite-wal

Environment

vCenter 6.x
vCenter 7.x

Cause

There are many possible causes for specific disk partitions running out of space. Some causes include:

• Partitions are not sized correctly.

• Large files in the partitions.

• Tables in the database consuming space.

• Services are failing to clean up files.

The files causing the high utilization can be identified by running the commands:
du -Shx / |sort -hr |head -20

and

find . -type f -print0 | xargs -0 du -h | sort -rh | head -n 10

Resolution

Note:
Please ensure that you have a valid backup or snapshot before continuing with the the steps below. For vCenter in Enhanced Link Mode, please reference this KB article:
VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

As identified, the following files need to be removed:

• /var/vmware/applmgmt/appliance_stats.sqlite
• /var/vmware/applmgmt/appliance_stats.sqlite-shm
• /var/vmware/applmgmt/appliance_stats.sqlite-wal

Please run the commands below in an ssh session connection to the vCenter:

1. cd /var/vmware/applmgmt
2. rm -f appliance_stats.sqlite*
3. service-control --stop statsmonitor
4. service-control --start statsmonitor