Troubleshooting vCenter Appliance /storage/log directory is 80% or more full
Article ID: 313077
Updated On:
Products
Issue/Introduction
Symptoms:
- vCenter Server Appliance (vCSA) reports errors similar to:
- vSphere UI Health Alarm", "Log disk exhaustion on vcenter name
- Database Health Alarm", "Core and Inventory Disk Exhaustion on vcenter name
- vCenter Server may be inaccessible with 503 Service Unavailable errors.
- vSphere Appliance Management Interface (VAMI) Monitor > Disk shows /storage/log at 75% full or more:
- At 75% of continuous disk use the space will trigger Yellow status warnings
- At 85% the space will trigger Red critical alarms
- At 95% the vpxd service will shut down to prevent file and database corruption
Environment
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server 7.0.x
VMware vCenter Server 8.0
Cause
- vCenter Server log bundles not being cleared after generation
- Very high frequency events filling up logs
- Services such as the Apache Tomcat Java Servlet service failing to clean up files
- /storage/log partition set too small
Resolution
To investigate the issue of vCenter Server /storage/space showing as full, follow these steps:
1. SSH into the vCenter Server appliance using an account with root privileges.
2. Navigate to the /storage/log directory:
# cd /storage/log 3. Run the following command to identify the largest files:
# find . -type f -print0 | xargs -0 du -h | sort -rh | head -n 10 This command will display the 10 largest files in the directory. Example output:
3.7G ./vmware/wcp/stdstream.log-2.stderr
2.5G ./vmware/wcp/stdstream.log-1.stderr
266M ./vmware/wcp/stdstream.log.stderr
190M ./vmware/vpxd/vpxd-profiler-154.log
104M ./vmware/procstate
101M ./vmware/vsphere-ui/logs/threadmonitor1.log
83M ./vmware/wcp/stdstream.log-4.stderr
46M ./vmware/vpxd-svcs/perf.log.37
45M ./vmware/sso/ssoAdminServer.log
41M ./vmware/vsphere-ui/logs/threadmonitor.log4. Run the following command to identify directories with a high number of files:
# find ./ -type d -exec sh -c 'echo -n "{}: " && find "{}" -type f | wc -l' \; | awk '$2 > 100' | sort -k2,2nr This command will display directories containing more than 100 files, sorted by file count. Example output:
./: 1442
./etc: 486
./etc/vmware: 366
./var: 321
./var/run: 275
./var/log: 274
./usr: 108 5. Analyze the output to identify large files or directories with numerous files that may be consuming excessive space.
6. Based on the results, verify whether one of these known issues is not causing the problem:
| Affected Versions | Associated Knowledgebase Article link |
|---|---|
| 6.0 | |
| 6.0 before Update 3, 6.5 before Update 1 | |
| 7.0 before Update 1C | |
| 7.0 before Update 3c | |
| 7.0 before Update 3o, 8.0 before Update 1 | |
| 7.0, 8.0, not resolved in any newer version | |
|
7.0 Update 1 through Update 2 |
|
| 7.0 Update 2. Fixed in Update 3 | |
| 8.0. Fixed in Update 3 | |
| All versions |
7. If the known issues don't identify the problem, and the search commands above don't provide helpful information, the drive might be too small for the current log traffic due to increased usage.
The size of the /storage/log virtual disk may be increased if the environment needs additional logging space. See "Resizing the vCenter virtual disks' in KB vCenter Server Appliance disk space is full (318953)
Additional Information
vCenter Server Appliance disk space is full (318953)
vCenter Server /storage/log filling up due to localhost_access.log and catalina.log in sso and lookupsvc log directories(318209)
Impact/Risks:
Under default settings, when the partition reaches 75% full of continuous disk use for at least 10 minutes:
- Disk usage alarm is being triggered.
When the partition reaches 95% full at any point
- The vCenter system will prevent the main vmware-vpxd service from starting to try to protect the file system from corruption.
Please keep in mind that this does not require it to be at 95% for a continuous period so the threshold might be crossed, shut down vpxd service, and then the usage on the disk may recede below the 95% level.
Deleting critical files may prevent the vCenter Server Appliance from working.
Resizing vCenter Appliance virtual disks carries the risk of data corruption.
WARNING: Ensure a good backup (VAMI file backup, VADP backup, or both) has recently been taken of the vCenter Appliance before deleting files or resizing the VCSA system's disks.
==============================================
In addition to using find command noted in the resolution section above, using the du (disk utilization) command below can assist to quickly identify large directories in a file system:
du -xh -d 1 | sort -rh
- Use this command and then 'walk' the file system folders with the 'cd' command to navigate into large sub directories.
- Re-run the du command the search for the source of large disk consumption and repeat using the command in sub directories as needed.
- OPTIONAL: Change the value of the '-d' option to increase the number of folder levels for the output report. Using the value '-d 1' will show one folder down form the current path. Reporting on 1 directory down from the current directory path allows quick evaluation of the command's output.
- Once directory navigation has ended in a directory with large disk utilization, run the command 'ls -lsah' to list files.
- Identify if there are large files or a large number of files causing the high disk space utilization and address as needed.
Feedback
