To investigate the issue of vCenter Server /storage/space showing as full, follow these steps:
1. SSH into the vCenter Server appliance using an account with root privileges.
2. Navigate to the /storage/log directory:
# cd /storage/log
3. Run the following command to identify the largest files:
# find . -type f -print0 | xargs -0 du -h | sort -rh | head -n 10
This command will display the 10 largest files in the directory. Example output:
3.7G ./vmware/wcp/stdstream.log-2.stderr
2.5G ./vmware/wcp/stdstream.log-1.stderr
266M ./vmware/wcp/stdstream.log.stderr
190M ./vmware/vpxd/vpxd-profiler-154.log
104M ./vmware/procstate
101M ./vmware/vsphere-ui/logs/threadmonitor1.log
83M ./vmware/wcp/stdstream.log-4.stderr
46M ./vmware/vpxd-svcs/perf.log.37
45M ./vmware/sso/ssoAdminServer.log
41M ./vmware/vsphere-ui/logs/threadmonitor.log
4. Run the following command to identify directories with a high number of files:
# find ./ -type d -exec sh -c 'echo -n "{}: " && find "{}" -type f | wc -l' \; | awk '$2 > 100' | sort -k2,2nr
This command will display directories containing more than 100 files, sorted by file count. Example output:
./: 1442
./etc: 486
./etc/vmware: 366
./var: 321
./var/run: 275
./var/log: 274
./usr: 108
5. Analyze the output to identify large files or directories with numerous files that may be consuming excessive space.
6. Based on the results, verify whether one of these known issues is not causing the problem:
Affected Versions | Associated Knowledgebase Article link |
---|---|
6.0 | |
6.0 before Update 3, 6.5 before Update 1 | |
7.0 before Update 1C | |
7.0 before Update 3c | |
7.0 before Update 3o, 8.0 before Update 1 | |
7.0, 8.0, not resolved in any newer version | |
7.0 Update 1 through Update 2. |
|
7.0 Update 1 and newer releases. Fixed in 8.0. | |
7.0 Update 2. Fixed in Update 3. | |
8.0. Fixed in Update 3. | |
All versions |
7. If the known issues don't identify the problem, and the search commands above don't provide helpful information, the drive might be too small for the current log traffic due to increased usage.
You might increase the size of the /storage/log virtual disk per vCenter Server Appliance disk space is full
vCenter Server Appliance disk space is full
vCenter Server /storage/log filling up due to localhost_access.log and catalina.log in sso and lookupsvc log directories
Impact/Risks:
Under default settings, when the partition reaches 75% full of continuous disk use for at least 10 minutes:
When the partition reaches 95% full at any point
Please keep in mind that this does not require it to be at 95% for a continuous period so the threshold might be crossed, shut down vpxd service, and then the use may recede,
Deleting critical files may prevent the vCenter Server Appliance from working.
Resizing vCenter Appliance virtual disks carries the risk of data corruption.
WARNING: Ensure you have a good backup of vCenter Appliance before deleting files or resizing disks.
==============================================
In addition to using find command noted in the resolution section above, using the du (disk utilization) command below can assist to quickly identify large directories in a file system:
du -xh -d 1 | sort -rh