ESXi hosts dropping vmsyslog logger message
search cancel

ESXi hosts dropping vmsyslog logger message

book

Article ID: 323601

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • In /var/log/vmkernel.log similar entries are seen:
2021-02-05T14:25:30.617Z cpu18:2097724)ALERT: vmsyslog logger <ip>:514 lost 443 log messages
2021-02-06T02:25:31.003Z cpu28:2097724)ALERT: vmsyslog logger <ip>:514 lost 900 log messages
2021-02-06T14:25:32.225Z cpu15:2097724)ALERT: vmsyslog logger <ip>:514 lost 605 log messages
2021-02-07T02:25:32.588Z cpu1:2097724)ALERT: vmsyslog logger <ip>:514 lost 139 log messages
2021-02-07T14:25:32.972Z cpu8:2097724)ALERT: vmsyslog logger <ip>:514 lost 235 log messages

 

  • In /var/log/.vmsyslogd.err similar entries are seen:
FileNotFoundError: [Errno 2] No such file or directory: 'logger -t Hostd < /var/run/vmware/hostdLogHeader.txt'
2021-02-18T00:11:18.481Z vmsyslog.loggers.file    : ERROR   ] Failed to write header on rotate. Exception: [Errno 2] No such file or di
rectory: 'logger -t Vpxa < /var/run/vmware/vpxaLogHeader.txt'
Traceback (most recent call last):
  File "/build/mts/release/bora-17325551/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmsyslog/loggers/file.py
", line 412, in writeLog
  File "/build/mts/release/bora-17325551/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmsyslog/loggers/file.py", line 358, in writeOnRotateHeader
FileNotFoundError: [Errno 2] No such file or directory: 'logger -t Vpxa < /var/run/vmware/vpxaLogHeader.txt'
2021-02-18T01:52:18.191Z vmsyslog.loggers.file    : ERROR   ] Failed to write header on rotate. Exception: [Errno 2] No such file or directory: 'logger -t Hostd < /var/run/vmware/hostdLogHeader.txt'
Traceback (most recent call last):
  File "/build/mts/release/bora-17325551/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmsyslog/loggers/file.py", line 412, in writeLog
  File "/build/mts/release/bora-17325551/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmsyslog/loggers/file.py", line 358, in writeOnRotateHeader
FileNotFoundError: [Errno 2] No such file or directory: 'logger -t Hostd < /var/run/vmware/hostdLogHeader.txt'
2021-02-18T05:00:46.210Z vmsyslog.loggers.file    : ERROR   ] Failed to write header on rotate. Exception: [Errno 2] No such file or directory: 'logger -t Hostd < /var/run/vmware/hostdLogHeader.txt'
Traceback (most recent call last):
  File "/build/mts/release/bora-17325551/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmsyslog/loggers/file.py", line 412, in writeLog
  File "/build/mts/release/bora-17325551/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmsyslog/loggers/file.py", line 358, in writeOnRotateHeader
FileNotFoundError: [Errno 2] No such file or directory: 'logger -t Hostd < /var/run/vmware/hostdLogHeader.txt'
2021-02-18T07:52:10.994Z vmsyslog.msgQueue        : ERROR   ] vrli.smn.dc:514 - lost 421486 log messages
2021-02-18T08:10:04.586Z vmsyslog.loggers.file    : ERROR   ] Failed to write header on rotate. Exception: [Errno 2] No such file or directory: 'logger -t Hostd < /var/run/vmware/hostdLogHeader.txt'
Traceback (most recent call last):
  File "/build/mts/release/bora-17325551/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmsyslog/loggers/file.py", line 412, in writeLog
  File "/build/mts/release/bora-17325551/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmsyslog/loggers/file.py", line 358, in writeOnRotateHeader

 

  • When checking the onrotate In  etc/vmsyslog.conf.d/ onrotate field is set to a file instead of a command, the below example Vsan,fdm,vdpi,vvold,vsfwd,cmmdsTimeMachine,rhttpproxy,rabbitmqproxy,hbrca,hostd,vpxa are affected.

grep -r 'onrotate*' etc/vmsyslog.conf.d/
etc/vmsyslog.conf.d/PyVsanMOProvider.conf:onrotate = printf '%%s - last log rotation time, %%s\n' "$(date --utc +%%FT%%T.%%3NZ)" "$(cat /var/run/vmware/vsansystemLogHeader.txt)" | logger -t vsansystem
etc/vmsyslog.conf.d/fdm.conf:onrotate = logger -t Fdm < /var/run/vmware/fdmLogHeader.txt
etc/vmsyslog.conf.d/vdpi.conf:onrotate = logger -t vdpi < /var/run/vmware/vdpiLogHeader.txt

etc/vmsyslog.conf.d/vvold.conf:onrotate = /var/run/vmware/vvoldLogHeader.txt

etc/vmsyslog.conf.d/vsfwd.conf:onrotate = logger -t vsfwd < /var/run/vmware/vsfwdLogHeader.txt
etc/vmsyslog.conf.d/cmmdsTimeMachine.conf:onrotate = logger -t cmmdsTimeMachine < /var/run/cmmdstimemachineLogHeader.txt

etc/vmsyslog.conf.d/rhttpproxy.conf:onrotate = /var/run/vmware/rhttpproxyLogHeader.txt

etc/vmsyslog.conf.d/cmmdsTimeMachineDump.conf:onrotate = logger -t cmmdsTimeMachineDump < /var/run/cmmdstimemachineDumpHeader.txt
etc/vmsyslog.conf.d/rabbitmqproxy.conf:onrotate = printf '%%s - last log rotation time, %%s\n' "$(date --utc +%%FT%%T.%%3NZ)" "$(cat /var/run/vmware/rabbitmqproxyLogHeader.txt)" | logger -t rabbitmqproxy 
etc/vmsyslog.conf.d/hbrca.conf:onrotate = /var/run/vmware/hbrcaLogHeader.txt
etc/vmsyslog.conf.d/hostd.conf:onrotate = logger -t Hostd < /var/run/vmware/hostdLogHeader.txt 
etc/vmsyslog.conf.d/vpxa.conf:onrotate = logger -t Vpxa < /var/run/vmware/vpxaLogHeader.txt



Environment

VMware vSphere ESXi 6.x

Cause

The file vmsyslog.conf.d should be displaying a file instead of a command. 

Resolution

Workaround:
1. Take a backup of the vmsyslog.conf.d folder.

2. Taking the vpxa log as an example the onrotate field 

Changed from 

onrotate = logger -t Vpxa < /var/run/vmware/vpxaLogHeader.txt

to 

onrotate = /var/run/vmware/vpxaLogHeader.txt

Also if /var/run/vmware/vpxaLogHeader.txt is not present it has to be created.

3. The above steps would need to repeat from the remaining logs files were onrotate field  are using a command 

4. After making configuration changes, load the new configuration by running this command:

esxcli system syslog reload

Powered by Wolken Software