vCenter log disk exhaustion or /storage/log full
Article ID: 313077
Updated On:
Products
Issue/Introduction
This article provides information on how to troubleshoot and resolve the /storage/log partition being full on vCenter Appliance.
Impact/Risks:
Under default settings, when the partition reaches 75% full of continuous disk use for at least 10 minutes:
Disk usage alarm is being triggered.Deleting critical files may prevent the vCenter Server Appliance from working.
Resizing vCenter Appliance virtual disks carries the risk of data corruption.
WARNING: Ensure a good backup (VAMI file backup, VADP backup, or both) has recently been taken of the vCenter Appliance before deleting files or resizing the VCSA system's disks.
Symptoms:
- vCenter Server Appliance (vCSA) reports errors similar to:
vSphere UI Health Alarm", "Log disk exhaustion on vcenter name.
Database Health Alarm", "Core and Inventory Disk Exhaustion on vcenter name- vCenter Server may be inaccessible with 503 Service Unavailable errors.
- vSphere Appliance Management Interface (VAMI) Monitor > Disk shows /storage/log at 75% full or more:
At 75% of continuous disk use the space will trigger Yellow status warnings
At 85% the space will trigger Red critical alarms- From VAMI Summary>Health Status> Storage(Alert)
From VAMI: Health Status: "File system /storage/log is low on storage space. Increase the size of disk /storage/log."- From the vCenter you may not be able to view vSAN related options like skyline health.
Environment
Cause
- vCenter Server log bundles not being cleared after generation
- Very high frequency events filling up logs
- Services such as the Apache Tomcat Java Servlet service failing to clean up files
- /storage/log partition set too small.
Resolution
To investigate the issue of vCenter Server /storage/space showing as full, follow these steps:
- SSH into the vCenter Server appliance using an account with root privileges.
- Navigate to the /storage/log directory:
cd /storage/log - Run the following command to identify the largest files:
find . -type f -print0 | xargs -0 du -h | sort -rh | head -n 10This command will display the 10 largest files in the directory. Example output:
3.7G ./vmware/wcp/stdstream.log-2.stderr 2.5G ./vmware/wcp/stdstream.log-1.stderr 266M ./vmware/wcp/stdstream.log.stderr 190M ./vmware/vpxd/vpxd-profiler-154.log 104M ./vmware/procstate 101M ./vmware/vsphere-ui/logs/threadmonitor1.log 83M ./vmware/wcp/stdstream.log-4.stderr 46M ./vmware/vpxd-svcs/perf.log.37 45M ./vmware/sso/ssoAdminServer.log 41M ./vmware/vsphere-ui/logs/threadmonitor.log - Run the following command to identify directories with a high number of files:
find ./ -type d -exec sh -c 'echo -n "{}: " && find "{}" -type f | wc -l' \; | awk '$2 > 100' | sort -k2,2nrThis command will display directories containing more than 100 files, sorted by file count. Example output:
./: 1442 ./etc: 486 ./etc/vmware: 366 ./var: 321 ./var/run: 275 ./var/log: 274 ./usr: 108 - Analyze the output to identify large files or directories with numerous files that may be consuming excessive space.
- Based on the results, verify whether one of these known issues is not causing the problem:
| Affected Versions | Associated Knowledgebase Article link |
|---|---|
| 6.0 | |
| 6.0 before Update 3, 6.5 before Update 1 | |
| 7.0 before Update 1C | |
| 7.0 before Update 3c | |
| 7.0 before Update 3o, 8.0 before Update 1 | |
| 7.0, 8.0, not resolved in any newer version | |
7.0 Update 1 through Update 2 | |
| 7.0 Update 2. Fixed in Update 3 | |
| 8.0. Fixed in Update 3 | |
| All versions |
Additional Information
vCenter Server Appliance disk space is full (318953)
vCenter Server /storage/log filling up due to localhost_access.log and catalina.log in sso and lookupsvc log directories(318209)
In addition to using find command noted in the resolution section above, using the du (disk utilization) command below can assist to quickly identify large directories in a file system:
du -xh -d 1 | sort -rh
- Use this command and then 'walk' the file system folders with the 'cd' command to navigate into large sub directories.
- Re-run the du command the search for the source of large disk consumption and repeat using the command in sub directories as needed.
- OPTIONAL: Change the value of the '-d' option to increase the number of folder levels for the output report. Using the value '-d 1' will show one folder down form the current path. Reporting on 1 directory down from the current directory path allows quick evaluation of the command's output.
- Once directory navigation has ended in a directory with large disk utilization, run the command 'ls -lsah' to list files.
- Identify if there are large files or a large number of files causing the high disk space utilization and address as needed.
Run df -i to check if the i nodes are full if you see any space related errors in the logs.
Sometimes storage/log may look fine when we run df -h but the storage/log could be full when you run df -i
cd /storage/logrun this query :
for i in *; do echo $i; find $i | wc -l; done | paste - - | sort -k2 -nYou may refer to this article
/storage/log filling up with imfile-state files | rsyslogd
If the known issues don't identify the problem, and the search commands above don't provide helpful information, the drive might be too small for the current log traffic due to increased usage.
The size of the /storage/log virtual disk may be increased if the environment needs additional logging space. See "Resizing the vCenter virtual disks' in KB vCenter Server Appliance disk space is full (318953)
Feedback
