High NSX logging volume, causing frequent log rollovers in NSX 4.x and ESXi 8.0

Products

• VMware NSX
• VMware vSphere ESXi 8.0

Issue/Introduction

• Excessive amount of logging data in the nsx-syslog log file in ESXi Hosts with NSX versions 4.x and ESXi versions 8.0
• NSX single log with embedded newline characters, broken into multiple lines.
• Above logging data is identifiable by the [+] sign in the log lines : /var/log/nsx-syslog.log

Examples:

yyyy-mm-ddThh:mm:ss.mssZ In(182) cfgAgent[2788014]: NSX 2788014 - [nsx@6876 comp="nsx-controller" subcomp="cfgAgent" tid="C0FC9C80" level="info"] Decoder: Received LOG_SWITCH_FIB VTEP old msg ########-####-####-####-############: vtep {
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] cfgAgent[2788014]: vtep_ip {
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] cfgAgent[2788014]: ipv4: ##.###.###.##
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] cfgAgent[2788014]: }
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] cfgAgent[2788014]: vtep_label {
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] cfgAgent[2788014]: label: 88065
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] cfgAgent[2788014]: }
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] cfgAgent[2788014]: segment_id {
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] cfgAgent[2788014]: ipv4: ##.###.###.##

yyyy-mm-ddThh:mm:ss.mssZ In(182) nestdb-server[2788056]: NSX 2788056 - [nsx@6876 comp="nsx-esx" subcomp="nsx-nestdb" tid="2788056" level="INFO"] ExtractRepeatedDelta (Patch Op): vtep {
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: self {
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: op: APPEND
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: }
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: }
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: mac {
yyyy-mm-ddThh:mm:ss.mssZ In(182) nestdb-server[2788056]: NSX 2788056 - [nsx@6876 comp="nsx-esx" subcomp="nsx-nestdb" tid="2788056" level="INFO"] ExtractRepeatedDelta (Patch): id {
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: left: ################
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: right: #####################
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: }
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: vtep {
yyyy-mm-ddThh:mm:ss.mssZ In(182)[+] nestdb-server[2788056]: vtep_ip {

Environment

vSphere ESXi 8.0
VMware NSX

Cause

Syslog service, vmsyslogd, will break messages into separate lines when it encounters embedded newlines in the log message. This is standard and uniform for all log messages, as embedded newlines are inferred as the need for parts to be printed in separate lines.

This was introduced for vmsyslogd using std::endl in ESXi 8.0 versions - Hence std::endl is translated to "" on ESXi host 7.0 but on ESXi host 8.0 it is changed to a newline resulting in a single log message being broken down into multiple log lines.

Resolution

This issue is resolved in NSX versions 4.2.1 or newer.

There is no workaround for prior NSX versions.

Additional Information

ESXi documentation on change in logging infrastructure for version 8.0:

https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-monitoring-and-performance-8-0/system-log-files/managing-esxi-log-files.html