VCSA inodes getting full due to large files queued for mails /var/spool/clientmqueue

Products

• VMware vCenter Server

Issue/Introduction

To fix the source of the problem which created a huge number of core.sendmail files in /storage/core and /var/spool/clientmqueue.

Symptoms:

• Unable to log into vCenter Web Client, /storage/core at high/100% utilization filled with core.sendmail.
• Filesystem utilization:

root@xxxx [ /storage/core ]# df -h
Filesystem                                Size  Used Avail Use% Mounted on
devtmpfs                                  7.9G     0  7.9G   0% /dev
tmpfs                                     7.9G   20K  7.9G   1% /dev/shm
tmpfs                                     7.9G  692K  7.9G   1% /run
tmpfs                                     7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda3                                  11G  5.7G  4.5G  56% /
tmpfs                                     7.9G  1.6M  7.9G   1% /tmp
/dev/mapper/netdump_vg-netdump            985M  1.3M  932M   1% /storage/netdump
/dev/sda1                                 120M   27M   87M  24% /boot
/dev/mapper/dblog_vg-dblog                 15G  1.9G   12G  14% /storage/dblog
/dev/mapper/imagebuilder_vg-imagebuilder  9.8G   23M  9.2G   1% /storage/imagebuilder
/dev/mapper/autodeploy_vg-autodeploy      9.8G   33M  9.2G   1% /storage/autodeploy
/dev/mapper/updatemgr_vg-updatemgr         99G  780M   93G   1% /storage/updatemgr
/dev/mapper/log_vg-log                    9.8G  1.8G  7.5G  19% /storage/log
/dev/mapper/core_vg-core                   50G   49G    1G  98% /storage/core
/dev/mapper/db_vg-db                      9.8G  1.5G  7.8G  16% /storage/db
/dev/mapper/seat_vg-seat                   25G  8.8G   15G  38% /storage/seat

• The below files are generated every 5 minutes under /storage/core:

-r-x------ 1 root      root       41M MM DD HH:MM core.sendmail.26320
-r-x------ 1 root      root       41M MM DD HH:MM core.sendmail.28221
-r-x------ 1 root      root       41M MM DD HH:MM core.sendmail.30153
-r-x------ 1 root      root       41M MM DD HH:MM core.sendmail.31610
-r-x------ 1 root      root       41M MM DD HH:MM core.sendmail.33434
-r-x------ 1 root      root       41M MM DD HH:MM core.sendmail.35229

• You might see the below errors in /var/log/vmware/messages:

YYYY-MM-DDZHH:MM:SS.234078+00:00 vcsa1 sendmail[4979]: x21B1Rqb037830: to=<[email protected]>, delay=HH:MM:SS, xdelay=00:00:00, mailer=esmtp, pri=120832, relay=yyyyyyy.com., dsn=4.0.0, stat=Deferred: Connection timed out with yyyyyyy.com.

 

Environment

VMware vCenter Server Appliance 6.5.x

Cause

• Open vCenter Web Client, verify Mail server settings:

                Select vCenter > Configure > General > Edit > Mail.

• The relay configured(relaymail.example,com) was not reachable as it was decommissioned.
•  As a result, a number of mails were queued as a SMTP relay was defined which VC was not able to communicate with.
• Due to large queued mails, sendmail service started crashing and filling /storage/core.

Resolution

• To stop the core.sendmail files from being generated, we need to stop the sendmail.service and reconfigure mail server relay.
• Run below command to stop the service:

                #systemctl stop sendmail.service

• Run below command to delete files in /var/spool/clientmqueue:

                #cd /var/spool/clientmqueue
                #echo * | xargs -n 100 rm

or

               #find /var/spool/clientmqueue/* -exec rm {} \;

• Above commands clear the entire folder to show zero contents eventually:

               # ls -ltrh /var/spool/clientmqueue/
                   total 0

• Confirm that no new core files were being created:

-r-x------ 1 root      root       41M MM DD HH:MM core.sendmail.35242

root@xxxx [ /storage/core ]# date
Day MM DD HH:DD:SS UTC YYYY

• Clear out the old core files.
  
  
• Final filesystem utilization:

========================
 # df -h

Filesystem                                Size  Used Avail Use% Mounted on
devtmpfs                                  7.9G     0  7.9G   0% /dev
tmpfs                                     7.9G   20K  7.9G   1% /dev/shm
tmpfs                                     7.9G  692K  7.9G   1% /run
tmpfs                                     7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda3                                  11G  5.7G  4.5G  56% /
tmpfs                                     7.9G  1.6M  7.9G   1% /tmp
/dev/mapper/netdump_vg-netdump            985M  1.3M  932M   1% /storage/netdump
/dev/sda1                                 120M   27M   87M  24% /boot
/dev/mapper/dblog_vg-dblog                 15G  1.9G   12G  14% /storage/dblog
/dev/mapper/imagebuilder_vg-imagebuilder  9.8G   23M  9.2G   1% /storage/imagebuilder
/dev/mapper/autodeploy_vg-autodeploy      9.8G   33M  9.2G   1% /storage/autodeploy
/dev/mapper/updatemgr_vg-updatemgr         99G  780M   93G   1% /storage/updatemgr
/dev/mapper/log_vg-log                    9.8G  1.8G  7.5G  19% /storage/log
/dev/mapper/core_vg-core                   50G   53M   47G   1% /storage/core
/dev/mapper/db_vg-db                      9.8G  1.5G  7.8G  16% /storage/db
/dev/mapper/seat_vg-seat                   25G  8.8G   15G  38% /storage/seat

 

Workaround:
If you are unable to pinpoint the source of the core.sendmail files, just stop the sendmail.service and clear out the core.sendmail files after clearing out the /var/spool/clientmqueue directory.

Additional Information

Impact/Risks:
vCenter vpxd service crashes and Web Client is inaccessible