ESXi syslog daemon logs "Dropping messages due to log stress" repeatedly

Products

• VMware vSphere ESXi

Issue/Introduction

This is article is to inform the reader that if too verbose of Distributed Firewall logging (DFW) is configured, the syslog daemon may run out of memory and restart periodically.

Symptoms:
Frequently the ESXi host syslog daemon (vmsyslogd) restarts, and produces log messages such as below:

/var/run/log/syslog.log:
2021-01-10T10:43:23.552Z cpu60:66165)MemSched: 14642: uw.66163 (1069) extraMin/extraFromParent: 64/64, logging (760) childEmin/eMinLimit: 21743/21760
2021-01-10T10:43:23.552Z cpu60:66165)MemSched: 14635: Admission failure in path: logging/vmsyslogd.66163/uw.66163

/var/log/.vmsyslogd.err
2021-01-10T10:43:25.246Z vmsyslog                 : CRITICAL] vmsyslogd daemon starting (1476814)
2021-01-10T10:43:25.458Z vmsyslog.main            : CRITICAL] Dropping messages due to log stress (qsize = 23531)
2021-01-10T10:49:35.553Z vmsyslog.msgQueue        : ERROR   ] 192.168.1.22:514 - lost 1 log messages
2021-01-10T11:08:12.984Z vmsyslog.loggers.file    : ERROR   ] Failed to write header on rotate. Exception: [Errno 2] No such file or directory: 'logger -t vsfwd < /var/run/vmware/vsfwdLogHeader.txt'

Environment

VMware ESXi 6.7.x
VMware ESXi 6.5.x

Cause

This is caused by too much logging generated by one or more virtual machine's NSX Firewall logging that is being passed to the ESXi syslog daemon. Too much logging can exhaust the ESXi syslog daemon and cause the service to restart.

Resolution

It is recommended to reduce/filter the amount of Firewall logging that is typically generated to avoid stressing the ESXi daemon. Per NSX documentation: To collect firewall audit logs on a syslog server, ensure that you have upgraded the syslog server to the recent version. Preferably, configure a remote syslog-ng server to collect the firewall audit logs.

For information on configuring/editing NSX Firewall logging, please refer to:
NSX Logging and System Events > Firewall Logs

Workaround:
A workaround may be to find the particular rule, or event that is causing a lot of log messages to be present, filter/adjust the rule and monitor the syslog daemon performance afterwards. Log directory paths can be found in the following documentation:
NSX Logging and System Events > Firewall Logs

Additional Information

Impact/Risks:
This can lead to potential loss of remote syslog messages being sent.