/storage/log becomes 100% full when exporting the vCenter Appliance log bundles

Products

• VMware vCenter Server 6.0
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

Symptoms:

•  The "/storage/log" partition becomes 100% full while trying to generate a vCenter Appliance log bundle.
  
  
•  This can be validated by running following command on vCenter. 

      # df -h 

           The output would be similar to the following :

     Filesystem             Size  Used  Avail Use%  Mounted on
     /dev/mapper/log_vg-log 9.9G  9.9G  0G    100%  /storage/log


• The size of the vCenter Server log bundle is relatively large, even when there are no hosts registered. For example, the generated log bundle was more than 5GB.
  
  
• Few vCenter management activities might be halted ( e.g. Storage vMotion / Compute vMotion ).

Environment

VMware vCenter Server 6.x
VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause

There are old support log bundles in the vCenter log directory which are consuming space or the new bundle size is fairly large. 

Resolution

To resolve this issue:

 

1. IMPORTANT! Snapshot the vCenter before doing any data removal procedures. See "Snapshot Best practices for vCenter Server Virtual Machines" and "VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice" for more information.

2. Connect to the vCenter Server Appliance using SSH and log in as the root user and enter shell by typing:  shell

3. Search for any old log bundles:  (Note: Log bundle names will appear by default with names like "vc-support-<vc_name>-<date&time>.tgz")

find / -iname *.tgz

4. Navigate to the relevant directory. (example for files in /storage/core/<filename>.tgz)

cd /storage/core

5. Delete the old log bundles: (Note: the preceding "./" restricts the rm to the current working directory)

rm ./*.tgz

6. Restart all vCenter Server services

service-control --stop --all && service-control --start --all

Note: Should these steps not be applicable to your specific scenario and no old log bundles can be found, please see vCenter Server Appliance disk space is full for additional issues related to insufficient disk space. 

Additional Information

Troubleshooting vCenter Appliance /storage/log directory is 80% or more full