High NSX logging volume causes frequent log rollovers and lost messages in NSX 4.x and ESXi 8.0

Products

• VMware NSX
• VMware vSphere ESXi 8.0

Issue/Introduction

• Excessive amount of logging data in the nsx-syslog log file in ESXi Hosts with NSX versions 4.x and ESXi versions 8.0
• NSX single log with embedded newline characters, broken into multiple lines.
• Above logging data is identifiable by the [+] sign in the log lines : /var/log/nsx-syslog.log

Examples:

yyyy-mm-ddThh:mm:ss.mssZ In(182) cfgAgent[2788014]: NSX 2788014 - [nsx@6876 comp="nsx-controller" subcomp="cfgAgent" tid="C0FC9C80" level="info"] Decoder: Received LOG_SWITCH_FIB VTEP old msg ########-####-####-####-############: vtep {
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] cfgAgent[2788014]: vtep_ip {
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] cfgAgent[2788014]: ipv4: ##.###.###.##
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] cfgAgent[2788014]: }
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] cfgAgent[2788014]: vtep_label {
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] cfgAgent[2788014]: label: 88065
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] cfgAgent[2788014]: }
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] cfgAgent[2788014]: segment_id {
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] cfgAgent[2788014]: ipv4: ##.###.###.##

yyyy-mm-ddThh:mm:ss.mssZ In(182) nestdb-server[2788056]: NSX 2788056 - [nsx@6876 comp="nsx-esx" subcomp="nsx-nestdb" tid="2788056" level="INFO"] ExtractRepeatedDelta (Patch Op): vtep {
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: self {
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: op: APPEND
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: }
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: }
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: mac {
yyyy-mm-ddThh:mm:ss.mssZ In(182) nestdb-server[2788056]: NSX 2788056 - [nsx@6876 comp="nsx-esx" subcomp="nsx-nestdb" tid="2788056" level="INFO"] ExtractRepeatedDelta (Patch): id {
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: left: ################
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: right: #####################
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: }
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: vtep {
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: vtep_ip {

• vmsyslogd might drop messages due to log stress.
  vmkernel.log
  yyyy-mm-ddThh:mm:ss.mssZ Al(177) vmkalert: cpu7:2097675)ALERT: vmsyslog logger <syslog server>:<port> lost NNNN log messages
  /var/log/.vmsyslogd.err
  yyyy-mm-ddThh:mm:ss.mssZ vmsyslog.main            : CRITICAL] Dropping messages due to log stress (qsize = 22500)
yyyy-mm-ddThh:mm:ss.mssZ vmsyslog.msgQueue        : ERROR   ] <syslog server>:<port> - lost 1 log messages
yyyy-mm-ddThh:mm:ss.mssZ vmsyslog.msgQueue        : ERROR   ] <syslog server>:<port> - lost NNNN log messages

Environment

vSphere ESXi 8.0

VMware NSX 4.x

Aria Operations for Logs 8.18.x

Cause

Syslog service, vmsyslogd, will break messages into separate lines when it encounters embedded newlines in the log message. This is standard and uniform for all log messages, as embedded newlines are inferred as the need for parts to be printed in separate lines.

This was introduced for vmsyslogd using std::endl in ESXi 8.0 versions - Hence std::endl is translated to "" on ESXi host 7.0 but on ESXi host 8.0 it is changed to a newline resulting in a single log message being broken down into multiple log lines.

Resolution

This issue is resolved in NSX versions 4.2.1 or newer, available at Broadcom Support Portal.
If you are having difficulty finding and downloading software, please review Download Broadcom products and software.

There is no workaround for prior NSX versions.

Additional Information

ESXi documentation on change in logging infrastructure for version 8.0 onwards:

Protocols, Formats and Framing of ESXi Syslog Messages

Note: You may observe incomplete log events in VMware Aria Operations for Logs due to the source NSX logging issue. This behavior does not originate from VMware Aria Operations for Logs, as the incomplete events are already broken into multiple lines within the /var/log/nsx-syslog.log file directly on the source ESXi host.