vCenter root partition filling up due to excessive logging in /var/mail/root file

Products

• VMware vCenter Server

Issue/Introduction

• The /var/mail/root file size in vCenter increases gradually and causes root partition to fill up to 100%.
• Below entries are seen in /var/mail/root file:
  
  considering file Path:/var/log/vmware/procstate-<date><time>.gz Size:2.3096351623535156 cTime: <date><time> mTime: <date><time> aTime: <date><time>
considering file Path:/var/log/vmware/messages.gz Size:0.3527231216430664 cTime: <date><time>mTime: <date><time> aTime: <date><time>file is older
considering file Path:/var/log/vmware/messages.gz Size:0.3422117233276367 cTime: <date><time>mTime: <date><time>aTime: <date><time>file is older
considering file Path:/var/log/vmware/messages.zst Size:0.7484207153320312 cTime: <date><time> mTime: <date><time> aTime: <date><time>
considering file Path:/var/log/vmware/procstate<date><time>.gz Size:2.3476810455322266 cTime: <date><time> mTime: <date><time> aTime: <date><time>file is older
  
  
• After performing a cleanup, this file resumes logging and progressively expands in size.

Environment

• vCenter 8.x
• vCenter 9.x

Cause

• This issue is caused by the space_utility cron job.
• Cron jobs capture the script outputs and emails it to the root user. These messages accumulate in the /var/mail/root file, causing it to expand continuously.
• /var/mail/root file shows the below contents:
  
  Loaded the configuration from /etc/vmware/appliance/space_utility.conf
Conf loaded from file
{"LOG_BASE_PATH": "/var/log/vmware/", "SUPPORT_BUNDLE_BASE_PATH": "/storage/log/", "CORE_DUMP_BASE_PATH": "/storage/core/", "RSYSLOG_BASE_PATH": "/etc/vmware-syslog", "REPORT_FILE_PATH": "/var/log/vmware/vcsa_space_util_report.json", "KNOWN_CORE_DUMP_CLEANUP_FILES": ["core.in:imuxsock", "core.in:imfile", "core.rs:main Q:Reg"],

Resolution

This is a known issue with vCenter and Broadcom Engineering is actively working towards fixing this issue in the future patch release.

Workaround

Cleanup the /var/mail/root file and edit the space_util.cron file to redirect the output of the cron job to /dev/null.

1. SSH to the vCenter as root user.
2. Remove the root file with the below command:
   
   rm /var/mail/root
   
   
3. Edit the file using the below command:
   
   vi /etc/cron.d/space_util.cron
   
   
4. Press Insert Key and add '> /dev/null 2>&1' to the end of the line:
   
   

   Default contents of the file:

   0 0 * * * root python3 /usr/lib/applmgmt/support/space_utility --report-file-path /var/log/vmware/ --report-file-name-base vcsa_space_util_report

   After modifying the file:

   0 0 * * * root python3 /usr/lib/applmgmt/support/space_utility --report-file-path /var/log/vmware/ --report-file-name-base vcsa_space_util_report > /dev/null 2>&1

5. Hit ESC key, type wq! and hit Enter key to save and exit the file.