A large number of log messages sent to the syslog server is lost with the following error message in vSAN environment : "'Issue detected on <ESXi hostname> in Datacenter: vmsyslog logger <syslog server IP address> lost ### log messages"

Products

• VMware vSphere ESXi

Issue/Introduction

In a vSAN environment, log entries or email alerts may indicate that syslog messages are being dropped.

• Following alert is observed in /var/run/log/vmkernel.log :

YYYY-MM-DDTHH:MM:SSZ Al(###) vmkalert: cpu##:#######)ALERT: vmsyslog logger <Syslog_server_name>:514 lost <No.of logs entries lost> log messages

• The log file /var/run/log/vmsyslog-dropped.log contains a high frequency of entries originating from vsandevicemonitord or cmmdsTimeMachineDump. Refer the log sample below:

YYYY-MM-DDTHH:MM:SSZ In(14) vsandevicemonitord[#######]: Exception getting SMART health status for vSAN disk <NAA_ID>

Environment

VMware vSAN
VMware vCenter Server 7.x
VMware vCenter Server 8.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

• The vmsyslogd daemon is unable to process the volume of logs generated by specific vSAN processes, specifically cmmdsTimeMachineDump and vsandevicemonitord.
• This excessive logging saturates the syslog buffer, leading to dropped packets before they can be transmitted to the remote syslog collector.

Resolution

To resolve this issue, implement syslog log filtering on the affected ESXi hosts to suppress the high-volume "TimeMachineDump" messages. Execute the following commands in succession via SSH on each ESXi host in the cluster:

1. Add the specific filter for TimeMachineDump: esxcli system syslog config logfilter add --filter="10|cmmdsTimeMachineDump|.*"

2. Enable the log filtering function: esxcli system syslog config logfilter set --log-filtering-enabled=true

3. Reload the syslog daemon to apply changes: esxcli system syslog reload

Additional Information

General log filtering information:

[NOTE: This is an example command and should be modified as per your use case]

• Add a log filter: esxcli system syslog config logfilter add -f="||"

Description:
  add                                                 Add a log filter.

Cmd options:
  -f|--filter=                                        The filter to be added. Format is: numLogs | ident | logRegexp. 'numLogs' sets
                                                        the maximum number of log entries for the specified log messages. After
                                                        reaching this number, the specified log messages are filtered and ignored.
                                                        'ident' specifies one or more system components to apply the filter to the log
                                                        messages that these components generate. 'logRegexp' specifies a case-sensitive
                                                        phrase with Python regular expression syntax to filter the log messages by
                                                        their content. (required)

For example, to filter out most messages from the cmmdsTimeMachine daemon after the tenth occurrence, use the following command: esxcli system syslog config logfilter add --filter="10|cmmdsTimeMachineDump|.*"

• Enable log filtering: esxcli system syslog config logfilter set --log-filtering-enabled=true

Cmd options:
  --log-filtering-enabled=                Enable or disable log filtering. (required)

• Reload vmsyslogd to load the new configuration: esxcli system syslog reload