Large numbers of vmsyslogger messages aren't being delivered.

Products

• VMware vSphere ESXi

Issue/Introduction

In a vSAN environment, you may receive email alerts or further see vmkernel logging showing the following:

2024-04-17T13:36:50.492Z Al(177) vmkalert: cpu34:2097996)ALERT: vmsyslog logger xx.xx.xx.xx:514 lost 2 log messages

Environment

vSphere vSAN
vSphere vCenter Server 7.x
vSphere vCenter Server 8.x
vSphere ESXi 7.x
vSphere ESXi 8.x

Cause

Vmsyslogd is unable to keep up with the amount of logging, and excessive logging appears to be coming from VSAN processes - cmmdTimeMachineDump and vsandevicemonitord.

On the ESXi host, the vmsyslog-dropped.log may be filled with messages from vsandevicemonitord or cmmdtimemachined

For example:

2024-04-24T08:08:17Z In(14) vsandevicemonitord[2102644]: Exception getting SMART health status for vSAN disk naa.####

Resolution

logfiltering enablement on cmmdsTimeMachineDump

The following three commands must be done on each host, in succession. 

• esxcli system syslog config logfilter add --filter="10|cmmdsTimeMachineDump|.*"
• esxcli system syslog config logfilter set --log-filtering-enabled=true
• esxcli system syslog reload

Additional Information

General log filtering information:

1) Add a log filter:

[NOTE: This is an example command and should be modified as per your use case]

$ esxcli system syslog config logfilter add -f="||"

Description:
  add                   Add a log filter.

Cmd options:
  -f|--filter=     The filter to be added. Format is: numLogs | ident | logRegexp. 'numLogs' sets
                        the maximum number of log entries for the specified log messages. After
                        reaching this number, the specified log messages are filtered and ignored.
                        'ident' specifies one or more system components to apply the filter to the log
                        messages that these components generate. 'logRegexp' specifies a case-sensitive
                        phrase with Python regular expression syntax to filter the log messages by
                        their content. (required)

For example, to filter out most messages from the cmmdsTimeMachine daemon after the tenth occurrence, use the following command: esxcli system syslog config logfilter add --filter="10|cmmdsTimeMachineDump|.*"

2 Enable log filtering:
$ esxcli system syslog config logfilter set --log-filtering-enabled=true

Cmd options:
  --log-filtering-enabled=
                        Enable or disable log filtering. (required)

3 Reload vmsyslogd to load the new configuration:

$ esxcli system syslog reload