VCSA inodes getting full due to large files queued for mails /var/spool/clientmqueue

Products

• VMware vCenter Server

Issue/Introduction

• Unable to log into vCenter Web Client, /storage/core at high/100% utilization filled with core.sendmail.
• vCenter vpxd service crashes and Web Client is inaccessible.
• Monitor alerts show /storage/core exceeding 95% threshold. vCenter remains accessible but disk pressure is critical.
• Take ssh to vCenter appliance and check Filesystem utilization using the command : df -h

Filesystem                                Size  Used Avail Use% Mounted on
devtmpfs                                  7.9G     0  7.9G   0% /dev
tmpfs                                     7.9G   20K  7.9G   1% /dev/shm
tmpfs                                     7.9G  692K  7.9G   1% /run
tmpfs                                     7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda3                                  11G  5.7G  4.5G  56% /
tmpfs                                     7.9G  1.6M  7.9G   1% /tmp
/dev/mapper/netdump_vg-netdump            985M  1.3M  932M   1% /storage/netdump
/dev/sda1                                 120M   27M   87M  24% /boot
/dev/mapper/dblog_vg-dblog                 15G  1.9G   12G  14% /storage/dblog
/dev/mapper/imagebuilder_vg-imagebuilder  9.8G   23M  9.2G   1% /storage/imagebuilder
/dev/mapper/autodeploy_vg-autodeploy      9.8G   33M  9.2G   1% /storage/autodeploy
/dev/mapper/updatemgr_vg-updatemgr         99G  780M   93G   1% /storage/updatemgr
/dev/mapper/log_vg-log                    9.8G  1.8G  7.5G  19% /storage/log
/dev/mapper/core_vg-core                   50G   49G    1G  98% /storage/core
/dev/mapper/db_vg-db                      9.8G  1.5G  7.8G  16% /storage/db
/dev/mapper/seat_vg-seat                   25G  8.8G   15G  38% /storage/seat

• The below files are generated every 5 minutes under /storage/core:

root filesize MM DD HH:MM core.sendmail.#####
root filesize MM DD HH:MM core.sendmail.#####
root filesize MM DD HH:MM core.sendmail.#####
root filesize MM DD HH:MM core.sendmail.#####
root filesize MM DD HH:MM core.sendmail.#####
root filesize MM DD HH:MM core.sendmail.#####

• You might see the below errors in /var/log/vmware/messages:

sendmail[]: to=<#######.com>, delay=HH:MM:SS, xdelay=00:00:00, mailer=esmtp, pri=120832, relay=#######.com., dsn=4.0.0, stat=Deferred: Connection timed out with #######.com.

 

Environment

VMware vCenter Server 8.x

Cause

• Open vCenter Web Client, verify Mail server settings: Select vCenter > Configure > General > Edit > Mail.
• The relay configured(relaymail.example,com) was not reachable as it was decommissioned.
• As a result, a number of mails were queued as a SMTP relay was defined which VC was not able to communicate with.
• Due to large queued mails, sendmail service started crashing and filling /storage/core.

Resolution

• To stop the core.sendmail files from being generated, we need to stop the sendmail.service and reconfigure mail server relay.
• Run below command to stop the service:
  systemctl stop sendmail.service
• Run below command to delete files in /var/spool/clientmqueue:
  cd /var/spool/clientmqueue
echo * | xargs -n 100 rm 
  OR
  find /var/spool/clientmqueue/* -exec rm {} \;
• Above commands clear the entire folder to show zero contents eventually and run the below command to check the same:
  ls -ltrh /var/spool/clientmqueue/
• Confirm that no new core files are being created:
  root filesize MM DD HH:MM core.sendmail.#####
root@#### [ /storage/core ]# date
Day MM DD HH:DD:SS UTC YYYY
• Clear out the old core files:
  rm -rf core.sendmail.*
• Verify vCenter Mail Settings: Log into the vSphere Web Client and navigate to vCenter > Configure > General > Edit > Mail. Ensure the SMTP server, sender address, and email addresses are correctly formatted and active.
• Run the below command to start the service: 
  systemctl start sendmail.service

Additional Information

• Workaround:
  • If you are unable to pinpoint the source of the core.sendmail files, just stop the sendmail.service and clear out the core.sendmail files, after clearing out the /var/spool/clientmqueue directory.
• Broadcom Contact Support