vCenter Services Fail to Start due to 100% inode usage on /storage/log

Products

• VMware vCenter Server

Issue/Introduction

vCenter services fail to start, and you'll find logs are not logging any new data when attempting to manually start services. 

Running df -h shows plenty of free space available. df -i will show all inodes for /storage/log in use. 

Environment

vCenter server 7.0

vcenter server 8.0

Cause

vCenter will warn of inode consumption and if ignored when fully consumed services will fail to start.

Resolution

To resolve this issue, please identify the log files consuming all the inodes. 

Use the following command on the /storage/log directory to reveal inode counts:

find . -maxdepth 3 -type d -exec sh -c '(echo -n "{}: "; find "{}" | wc -l)' \;

This will output an inode count. There is going to be one path with an extremely high inode count. Check the contents of that directory. You will likely find thousands (tens of thousands) of 0b log files present. Remove the 0b files to free up the inode space.

Use the command below to identify the directories with the largest number of files.

find ./ -type d -exec sh -c 'echo -n "{}: " && find "{}" -type f | wc -l' \; | awk '$2 > 100' | sort -k2,2nr 

Once the directories with a large number of files have been identified, check and remove the stale or old files to reduce the number of files.

Additional Information

As an example I had an issue where there were tens of thousands of vmware-vsan-health-service-xxxxx.log.gz files 0b in size consuming all the inodes. The following command removed them:

find . -name 'vmware-vsan-health-service-*.log.gz' | xargs rm -f