ESX host stops logging into vmkernel logs

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• You see that there are no messages logged in vmkernel log on the ESX host
• The vmkernel stops logging in the /var/log file of the ESX host
• /var/log directory becomes 100% full

Environment

VMware ESX Server 3.5.x
VMware ESX 4.0.x
VMware ESX 4.1.x

Resolution

To resolve this issue, you must restart vmkernel logging.

To restart vmkernel logging:

1. Run this command to verify if there is sufficient free space available for logging:
   
   #df -h
   
   
2. Run this command to verify if the syslog service is running:
   
   #service syslog status
   
   
3. Run this command to restart the syslog service:
   
   #service syslog restart
   
   
4. Run this command to verify if the vmklogger and vmklogger watchdog processes are running:
   
   #ps -aux | grep vmklogger
   
   If these services are running, you must see an output similar to:
   
   /bin/sh /usr/bin/vmware-watchdog -s vmklogger /usr/sbin/vmklogger
   /usr/sbin/vmklogger
   
   
5. If the vmklogger watchdog process is not running, run this command to start it:
   
   #/usr/bin/vmware-watchdog -s vmklogger /usr/sbin/vmklogger &
   
   
6. If the vmklogger watchdog process is running, run this command to restart the vmklogger process:
   
   #killall -HUP vmklogger
   
   
7. Check if the /var partition is full, it will show 100% used. You can find out the top 30 consumers of the space on this /var partition by giving the command
   
   du -a /var | sort -n -r | head -n 30
   
   If you find the files such as "nnn.core" in the /var/core/ partition, that can be deleted to release sufficient space to make the ESX host responsive again
   
   
8. After the space has been released, restart the management service.
   
   service mgmt-vmware restart