Syslog message loss on ESXi hosts with error "vmsyslog logger :514 lost #### log messages

Products

• VMware vSphere ESXi

Issue/Introduction

• Below errors would be noticed in /var/run/log/hostd.log 
  

  <timestamps> In(166) Hostd[264560]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 9507 : Issue detected on <esxi-fqdn/ip> in ha-datacenter: vmsyslog logger <syslog-server-ip>:514 lost 228556 log messages
  <timestamps> In(166) Hostd[264566]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 9520 : Issue detected on <esxi-fqdn/ip> in ha-datacenter: vmsyslog logger <syslog-server-ip>:514 lost 217627 log messages

  
  
  
• Running below command confirms the syslog settings are in place. Example output below :  
  esxcli system syslog config get



  [root@esxi] esxcli system syslog config get
     Allow Vsan Backing: false
     Check Certificate Revocation List: false
     Dropped Log File Rotation Size: 100
     Dropped Log File Rotations: 10
     Enforce SSLCertificates: true
     Local Log Output: /scratch/log
     Local Log Output Is Configured: false
     Local Log Output Is Persistent: true
     Local Logging Default Rotation Size: 1024
     Local Logging Default Rotations: 8
     Log Level: error
     Log To Unique Subdirectory: false
     Message Queue Drop Mark: 90
     Remote Host: tcp://<syslog-ip>:514
     Remote Host Connect Retry Delay: 180
     Remote Host Maximum Message Length: 4096
     Strict X509Compliance: false

  
  
  
  
• Testing the port is reachable from the ESXi host SSH was unsuccessful :
  nc -z <syslogserver> 514



  [root@esxi:~] nc -z <syslog-server-ip> 514   >> SUCCESSFULL
  Connection to <syslog-server-ip> 514 port [tcp/shell] succeeded!

  [root@esxi:~] nc -z <syslog-server-ip> 514   >> PORT NOT OPEN OR UNSUCCESSFUL CONNECTION

• Packet capture on ESXi showed no packets being sent outside ESXi.
pktcap-uw --uplink vmnic# --dir 2 -o - | tcpdump-uw -enr - | grep -i <syslog-server-ip>

Environment

VMware Cloud Foundation (VCF)
VMware Aria Operations for Logs
VMware ESXi

Cause

The ESXi outbound syslog firewall rule is disabled, which explicitly blocks the vmsyslogd daemon from transmitting logging traffic (UDP/TCP 514 or TCP 1514) across the host network boundary to the remote destination.

Resolution

Procedure to enable Syslog Firewall Rule via ESXi Host Client UI 

• Access the ESXi Host Client by navigating to https://<ESXi-FQDN-or-IP>/ui in a supported web browser.

• Authenticate using root or an account with equivalent administrative privileges.

• In the left Navigator pane, click Networking.

• In the main configuration pane, click the Firewall rules tab.

• Locate the syslog rule in the displayed list.

• Select the syslog rule to highlight it.

• Click the Actions dropdown menu (or right-click the highlighted rule).

• Select Enable.

Additional Information

• To open outbound traffic through the ESXi Firewall on UDP port 514 and TCP ports 514 and 1514:
  esxcli network firewall ruleset set --ruleset-id=syslog --enabled=true
esxcli network firewall refresh
  
  
• To test if the syslog server is receiving messages from ESXi host:
  esxcli system syslog mark --message "Syslog Test Message"


• Configuring syslog on ESXi