ESXi syslog fails with "Issue detected on <esxi-fqdn/ip> in ha-datacenter: vmsyslog logger <syslog-server-ip>:514 lost #### log messages"

Products

• VMware vSphere ESXi

Issue/Introduction

• Below errors would be noticed in hostd.log under /var/log/ or /var/run/log/ : 
  

  <timestamps> In(166) Hostd[264560]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 9507 : Issue detected on <esxi-fqdn/ip> in ha-datacenter: vmsyslog logger <syslog-server-ip>:514 lost 228556 log messages
  <timestamps> In(166) Hostd[264566]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 9520 : Issue detected on <esxi-fqdn/ip> in ha-datacenter: vmsyslog logger <syslog-server-ip>:514 lost 217627 log messages

  
  
  
• Running below command confirms the syslog settings are in place: 
  esxcli system syslog config get



  [root@esxi] esxcli system syslog config get
     Allow Vsan Backing: false
     Check Certificate Revocation List: false
     Dropped Log File Rotation Size: 100
     Dropped Log File Rotations: 10
     Enforce SSLCertificates: true
     Local Log Output: /scratch/log
     Local Log Output Is Configured: false
     Local Log Output Is Persistent: true
     Local Logging Default Rotation Size: 1024
     Local Logging Default Rotations: 8
     Log Level: error
     Log To Unique Subdirectory: false
     Message Queue Drop Mark: 90
     Remote Host: tcp://<syslog-ip>:514
     Remote Host Connect Retry Delay: 180
     Remote Host Maximum Message Length: 4096
     Strict X509Compliance: false

  
  
  
  
• Testing the port is reachable from the ESXi host SSH was unsuccessful :
  nc -z <syslogserver> 514



  [root@esxi:~] nc -z <syslog-server-ip> 514   >> PORT NOT OPEN OR UNSUCCESSFUL CONNECTION
  [root@esxi:~] nc -z <syslog-server-ip> 514   >> SUCCESSFULL
  Connection to <syslog-server-ip> 514 port [tcp/shell] succeeded!

• Packet capture on ESXi showed no packets being sent outside ESXi.
pktcap-uw --uplink vmnic# --dir 2 -o - | tcpdump-uw -enr - | grep -i <syslog-server-ip>

Environment

VMware ESXi
Syslog Server

Cause

ESXi was not sending packets out due to the syslog firewall rules being disabled.

Resolution

Enable the syslog firewall rule on ESXi : ESXi UI > Networking > Firewall rules

Additional Information

• To open outbound traffic through the ESXi Firewall on UDP port 514 and TCP ports 514 and 1514:
  esxcli network firewall ruleset set --ruleset-id=syslog --enabled=true
esxcli network firewall refresh
  
  
• To test if the syslog server is receiving messages from ESXi host:
  esxcli system syslog mark --message "Syslog Test Message"

• Configuring syslog on ESXi