ESXi syslog daemon logs "Dropping messages due to log stress" repeatedly

Products

• VMware vSphere ESXi

Issue/Introduction

This is article is to inform the reader that if too verbose of Distributed Firewall logging (DFW) is configured, the syslog daemon may run out of memory and restart periodically.

Symptoms:
Frequently the ESXi host syslog daemon (vmsyslogd) restarts, and produces log messages such as below:

/var/run/log/syslog.log:
YYYY-MM-DDTHH:MM:SS.XXXZ cpu60:66165)MemSched: 14642: uw.66163 (1069) extraMin/extraFromParent: 64/64, logging (760) childEmin/eMinLimit: 21743/21760
YYYY-MM-DDTHH:MM:SS.XXXZ cpu60:66165)MemSched: 14635: Admission failure in path: logging/vmsyslogd.66163/uw.66163

/var/log/.vmsyslogd.err
YYYY-MM-DDTHH:MM:SS.XXXZ vmsyslog                 : CRITICAL] vmsyslogd daemon starting (1476814)
YYYY-MM-DDTHH:MM:SS.XXXZ vmsyslog.main            : CRITICAL] Dropping messages due to log stress (qsize = 23531)
YYYY-MM-DDTHH:MM:SS.XXXZ vmsyslog.msgQueue        : ERROR   ] XX.XX.XX.XX:514 - lost 1 log messages
YYYY-MM-DDTHH:MM:SS.XXXZ vmsyslog.loggers.file    : ERROR   ] Failed to write header on rotate. Exception: [Errno 2] No such file or directory: 'logger -t vsfwd < /var/run/vmware/vsfwdLogHeader.txt'

Environment

VMware ESXi 6.7.x
VMware ESXi 6.5.x

Cause

This is caused by too much logging generated by one or more virtual machine's NSX Firewall logging that is being passed to the ESXi syslog daemon. Too much logging can exhaust the ESXi syslog daemon and cause the service to restart.

Resolution

It is recommended to reduce/filter the amount of Firewall logging that is typically generated to avoid stressing the ESXi daemon. Per NSX documentation: To collect firewall audit logs on a syslog server, ensure that you have upgraded the syslog server to the recent version. Preferably, configure a remote syslog-ng server to collect the firewall audit logs.

For information on configuring/editing NSX Firewall logging, please refer to:
Configure Remote Logging

Workaround:
A workaround may be to find the particular rule, or event that is causing a lot of log messages to be present, filter/adjust the rule and monitor the syslog daemon performance afterwards. Log directory paths can be found in the following documentation:
Configure Remote Logging

Additional Information

Impact/Risks:
This can lead to potential loss of remote syslog messages being sent.