ESXi host logs not updating despite no errors observed

Products

• VMware vSphere ESXi

Issue/Introduction

• Log files such as vmkernel.log and vobd.log are not being updated with current logging data.
• The ESXi host is not transmitting logs to any configured syslog server like VMware Aria Operations for Logs.
• The following error in the hidden log .vmsyslogd.err located in /var/log/. Since this is a hidden log can be observed, it will not be visible with a standard ls command. To view it, please use ls -la within the /var/log/ directory to list all files, including hidden ones.:

  YYYY-MM-DDTHH:MM vmsyslog : CRITICAL] vmsyslogd daemon starting (65992)
YYYY-MM-DDTHH:MM vmsyslog.main : ERROR ] switching to normal mode from init mode
YYYY-MM-DDTHH:MM vmsyslog.main : CRITICAL] Dropping messages due to log stress (qsize = 25000)

Cause

This issue can occur due to change which is done on the environment where attempts were made at increasing or decreasing the logging level.

Resolution

• Connect to the ESXi host via SSH
• Check the watchdog and vmsyslogd process IDs (PIDs)by running the command:

ps -Cc|grep vmsyslogd

• You see output similar to:

1234 1234  vmsyslogd  /bin/python /usr/lib/vmware/vmsyslog/bin/vmsyslogd.pyc -i

5678 5678  wdog-5678  /bin/python /usr/lib/vmware/vmsyslog/bin/vmsyslogd.pyc -i

• Run the kill command for the watchdog process first and then the vmsyslogd process: 

kill -9 5678

kill -9 1234

• Restart the vmsyslogd process by running the command:

/usr/lib/vmware/vmsyslog/bin/vmsyslogd

or

/etc/init.d/vmsyslogd restart

Additional Information