Unable to collect vCenter Server log bundles due to /storage/log full

Products

• VMware vCenter Server

Issue/Introduction

• Failed to collect log bundles from vCenter Server using the vSphere Client.
• Alternatively, a vCenter Server support bundle was generated but is sized <100B and does not contain any of the expected files or vCenter logs. 
• The output of 'df -h' ran on the vCenter shows the /storage/log location with high usage (>75% usuage) or low available space (i.e. <3 or 4 GB free space).

Environment

• VMware vCenter Server 7.x
• VMware vCenter Server 8.x

Cause

When collecting log bundles from vSphere Client, log bundles are generated in the /storage/log.
Therefore, the log bundle generation process fails due to insufficient space in /storage/log.

Resolution

Workarounds

Generate the support bundle in a different partition

• If /storage/log does not have sufficient space, can collect log bundles using the following method to have it generate in another file system location with more available space. 

GUI

1. Access the following URL in your browser
   https://<vCenter Server FQDN>/appliance/support-bundle
   
2. Enter the root credentials for the vCenter Server Appliance.
3. Verify the download in your browser.

CLI

The vc-support command generates log bundles in /storage/log by default, but you can specify the destination for log bundle generation using the -w option.

1. Log in to vCSA using SSH
2. Enter the shell command to access the Bash shell.
   shell
3. Check the partition with the most free space (e.g., storage/core)
   df -h
4. Specify a partition with free space to collect log bundles
   vc-support -w /storage/<partition name>
   Example command
   vc-support -w /storage/core
5. Download log bundle

Note: If you cannot to download log bundles using WinSCP or similar tools, please refer to Connecting to vCenter Server Virtual Appliance using WinSCP fails with the error: Received too large (1433299822 B) SFTP packet. Max supported packet size is 1024000 B.

Clean up aged core dump files in /storage/core

• A vCenter Server support bundle includes any core.* files located in /storage/core. 
  • As these core dump files can accumulate over time, there may be many GB worth of these files present in /storage/core. 
  • Cleaning up older (created more than >1 month ago) core dump files from /storage/core will often significantly reduce the overall size of the vCenter Server support bundle and potentially allow the support bundle to properly generate. 

Increase size of /storage/log partition

• vCenter Servers installed on older versions like 6.x then upgraded to 8.x often have /storage/log file system sized smaller than the 50GB that is standard for vCenter 8.x. 
  • In this event it may be necessary to increase the size of /storage/log to 50GB to provide enough space for vCenter Server support bundles. See "The log partition size on this vCenter Server is smaller than the recommended 50 GB", increase the disk space of /storage/log to 50GB on vCenter 8.0 for details on this, as well as for steps to increase the log partition. 

 

Additional Information

See vCenter log disk exhaustion or /storage/log full for assistance troubleshooting logging and log partition space issues.