VMware ESXi host stops sending syslogs to remote server

Products

Issue/Introduction

• An ESXi host stops sending logs to a remote syslog server.
   
• You see the event message:
  
  esx.problem.vmsyslogd.remote.failure
   
• In the /var/log/vobd.log file, you see entries similar to:
  
  [UserLevelCorrelator] nnnnnnnnus: [vob.user.vmsyslogd.remote.failure] The host "x.x.x.x:514" has become unreachable. Remote logging to this host has stopped.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi 7.0 and later.

Cause

This event indicates that the ESXi syslog service vmsyslogd stopped sending messages to a remote syslog server. This issue may occur if:

• Network connection is interrupted 
• Remote host closed the connection abruptly.
• A firewall is blocking the Port used for Syslog Service communication
• The remote syslog server is offline

Resolution

Reason for disconnection

1. Review the log file /var/log/.vmsyslogd.err on the ESXi host to identify the timeframe and any cited reason for connection failure from the perspective of the ESXi host vmsyslogd service.
   
   
2. Determine whether the ESXi host is correctly configured to send logs to the remote syslog server. 
    
3. Determine whether the remote syslog server logs agree with the timeframe for the connection failure. Identify any cited reason for the connection failure from the perspective of the remote syslog server.
    
4. Determine whether the network connection path between the ESXi host and the remote syslog server experienced a temporary or permanent interruption.
    
5. Determine whether the ESXi host logs have resumed flowing from to the remote syslog server automatically.
    

Automatic Reconnection

In some versions of vSphere ESXi, the syslog service automatically reconnects to the remote syslog server after a network interruption.
 

Manual Reconnection

If the vSphere ESXi syslog service does not automatically re-establish the connection to the remote syslog server, reconnect manually.

To reconnect manually:

1. Open the local ESXi Shell i.e. SSH to the ESXi .  
2. Reload the syslog server by running the command: # esxcli system syslog reload
   Note: If the esxcli system syslog reload command returns the error Failed to signal reload to vmsyslogd, then the vmsyslogd process is likely not running. Continue with steps in the Starting Service section.
    

Starting the service

If the vSphere ESXi syslog service is not running correctly or has exited, start the service.

To start the syslog service:

1. Open the local ESXi Shell on the affected ESXi host.
2. Validate that the syslog service daemon vmsyslogd is not running by running the command: # ps -Cuv | grep vmsyslogd 
3. If the command does not return a result, start the syslog service daemon by running the command: # /usr/lib/vmware/vmsyslog/bin/vmsyslogd 
4. Reload the syslog configuration by running the command: # esxcli system syslog reload