vCenter Services Fail to Start due to 100% inode usage on /storage/log

Products

• VMware vCenter Server

Issue/Introduction

vCenter services fail to start, and you'll find logs are not logging any new data when attempting to manually start services. 

Running df -h shows plenty of free space available. df -i will show all inodes for /storage/log in use. 

Cause

vCenter will warn of inode consumption and if ignored when fully consumed services will fail to start.

Resolution

To resolve this you will need to identify the log files consuming all the inodes. 

Use the following command on the /storage/log directory to reveal inode counts:

find . -maxdepth 3 -type d -exec sh -c '(echo -n "{}: "; find "{}" | wc -l)' \;

This will output an inode count. There is going to be one path with an extremely high inode count. Check the contents of that directory. You will likely find thousands (tens of thousands) of 0b log files present. Remove the 0b files to free up the inode space.

Additional Information

As an example I had an issue where there were tens of thousands of vmware-vsan-health-service-xxxxx.log.gz files 0b in size consuming all the inodes. The following command removed them:

find . -name 'vmware-vsan-health-service-*.log.gz' | xargs rm -f