ESXi log Events Appearing Out of Chronological Order in Syslog server

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

• Log events from VMware ESXi hosts appear in the Syslog server (e.g., IBM QRadar) out of chronological order.
• Inconsistencies between the 'Log Time' (timestamp in the log message) and 'Storage Time' (time received by the Syslog server).
• Syslog messages may appear truncated or split into multiple entries.

Environment

ESXi 8.x

Cause

This behavior is typically caused by the use of the UDP protocol for syslog forwarding:

1. UDP Sequencing: As a connectionless protocol, UDP does not guarantee packet delivery or the sequence in which packets arrive at the destination. In high-traffic environments or networks with latency, packets can arrive at the syslog server out of order.
2. ESXi Timezone: ESXi transmits all syslog data exclusively in UTC. If the Syslog is configured to use its local receipt time for sorting rather than parsing the UTC timestamp within the message, logs may appear shifted or misordered.

Resolution

To ensure reliable and chronological log ingestion, reconfigure the ESXi hosts to use TCP or SSL/TLS for syslog forwarding.

1. Reconfigure Syslog Protocol to TCP:

• esxcli system syslog config set --loghost='tcp://<syslog-server-ip>:514'
• esxcli system syslog reload
  Note: If using a non-standard port on ESXi 8.0 Update 2b or later, the firewall will automatically open the port. On earlier versions, a custom firewall rule may be required.

2. Configure Syslog Parsing: 

 Ensure that the Syslog is configured to:

• Parse the timestamp inside the syslog header rather than using the arrival time.
• Handle UTC-to-Local conversion on the server side, as ESXi cannot be configured to send logs in a local timezone. ESXi Timezone Support 

 

Additional Information

Configuring syslog on ESXi
ESXi Timezone Support