Syslog logger dropping log messages. rMinPeak growing above eMinPeak for logging group

Products

• VMware vSphere ESXi

Issue/Introduction

To stop the out of memory condition for logging group.

Symptoms:

• ESXi Host is reporting an alarm

Alarm 'Host error' on <Host Name> triggered by event 30258576 'Issue detected on <Host Name> in <Cluster Name>: vmsyslog logger <ip>:514 lost 2210935 log messages

• In /var/log/vmkernel.log similar entries are seen

 2020-04-09T17:25:34.490Z cpu24:65990)MemSched: 14635: Admission failure in path: logging/vmsyslogd.4810784/uw.4810784
2020-04-09T17:28:14.367Z cpu29:65990)MemSched: 14635: Admission failure in path: logging/vmsyslogd.4810998/uw.4810998
2020-04-09T17:41:33.482Z cpu0:65990)MemSched: 14635: Admission failure in path: logging/vmsyslogd.4811898/uw.4811898

2020-04-08T13:33:12.334Z cpu0:65990)ALERT: vmsyslog logger <IP address>:514 lost 2210935 log messages

• In /var/log/.vmsyslogd.err similar entries are seen

Traceback (most recent call last):
  File "/build/mts/release/bora-15256549/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmsyslog/loggers/file.py", line 379, in writeLog
  File "/build/mts/release/bora-15256549/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 676, in __init__
  File "/build/mts/release/bora-15256549/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 1228, in _execute_child
OSError: [Errno 28] No space left on device


Note: The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment.

Cause

vmsyslog is spawning subprocess frequently to execute this on-rotate command. This is causing memory exhaustion, resulting in  memory limit reaching the max configured for ‘logging’ group.

Resolution

This issue is resolved in

• VMware vSphere ESXi 6.7 Patch release ESXi670-202004002
• VMware vSphere ESXi 6.5 Patch release ESXi650-202007001

These patches are available to download at Customer Connect Patch Downloads

Workaround:
To workaround this issue

1. Verify the memory limits:
# localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group getmemconfig -g host/vim/vmvisor/logging

Example
rMinPeak Is greater than eMinPeak for logging:

gid name parGid eMinPeak rMinPeak
752 logging 14 87040 94952
14 vmvisor 2 2999688 2999688


2.Increase the memory allocation for the pool logging to maximum  120mb using following command.

localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group setmemconfig -g host/vim/vmvisor/logging -i 82 -m 120 -u mb


Note: The workaround is not persistent across reboot since /usr/lib/vmware/rp/bin/configRP script is called after reboot which will restore default configs as default values are hardcoded which cannot be modified.