Loss of log messages with errors "vmsyslog logger X.X.X.X lost xxx log messages"
book
Article ID: 334578
calendar_today
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
Symptoms:
In /var/log/vmkernel.log you see errors similar to:
2019-01-02T05:29:43.230Z cpu63:66187)ALERT: vmsyslog logger 1xx.1xx.2xx.39:514 lost 83302 log messages
In /var/log/.vmsyslogd.err you see errors similar to:
2019-01-02T05:29:43.377Z vmsyslog.loggers.file : ERROR ] Failed to spawn onrotate call Traceback (most recent call last): File "/build/mts/release/bora-10719125/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmsyslog/loggers/file.py", line 379, in writeLog File "/build/mts/release/bora-10719125/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 676, in __init__ File "/build/mts/release/bora-10719125/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 1221, in _execute_child OSError: [Errno 28] No space left on device 2019-01-02T05:32:13.804Z vmsyslog : CRITICAL] vmsyslogd daemon starting (87201097) 2019-01-02T05:32:14.005Z vmsyslog.main : CRITICAL] Dropping messages due to log stress (qsize = 25000)
Note:The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment.
Environment
VMware vSphere ESXi 6.5
Cause
If the network is slow, the write thread to network buffer will be faster than the reader thread which leads to losing messages.
Resolution
This issue is resolved in VMware vSphere ESXi 6.5 U3 and ESXi 6.7 U3 release.