ESXi vmsyslog drops messages occasionally when the network message buffer is full

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Messages in /var/run/log/vmkernel.log indicate that entries are being dropped

2022-02-22T02:28:13.752Z cpu87:2097818)ALERT: vmsyslog logger 10.10.10.10:514 lost 1 log messages

• Messages in /var/run/log/vmsyslogd.err show that  

2022-02-22T02:28:13.944Z vmsyslog.msgQueue        : ERROR   ] 10.10.10.10:514 - lost 751 log messages

• vCenter "Host Error" alarms are triggered on hosts where vmsyslog reports that it "lost xxx log messages"

Environment

VMware vSphere ESXi 7.0

Cause

This can occur when the network message buffer is filled up on ESXi and causes the messages to be dropped even though the logging volume is within the limits that vmsyslogd can handle.

Resolution

This is resolved in ESXi 7.0 Update 3o (Build 22348816)

Workaround:

1. Remove current remote syslog configuration
2. Reboot the ESXi host
3. Recreate the syslog configuration
4. Reboot again
5. Check functionality and logs at this point for reoccurrence.

 

It is also possible to disable or edit the "Host Error" alarm in vCenter for these events.

See Create or Edit Alarms for more information on editing alarms.