Truncated NSXT-IDPS logs in ESXi
Article ID: 345418
Updated On:
Products
Issue/Introduction
This article provides information regarding the default syslog message length present in ESXi host, and a way to change the same within the ESXi by manipulating the Syslog.global.remoteHost.maxMsgLen parameter.
Symptoms:
The NSXT-IDPS logs get truncated when sent from the ESXi host to any syslog server.
In the below IDPS event log, we can see the 1 IDPS event is getting broken down into 3 subsequent Syslog messages:
2023-09-25T16:07:57Z IDPS-EVT: [2105786]: {"timestamp":"2023-09-25T16:07:22.938586+0000","flow_id":8771907372xxxx,"pcap_cnt":89695629,"event_type":"alert","src_ip":"10.16.x.x","src_port":57762,"dest_ip":"20.x.x.x","dest_port":443,"proto":"TCP","direction":"to_server","metadata":{"flowbits":["LL.verifier_tcp_successful","LL.verifier_tcp_failed","LL.verifier_tcp_blocked"]},"nsx_metadata":{"flow_src_ip":"10.16.x.x","flow_dest_ip":"20.x.x.x","flow_dir":2,"rule_id":xxxx,"profile_id":"8e26cd1f-d749-4aa9-a2ee-xxxxxxxx","user_id":0,"vm_uuid":"5037439c-xxxx-1287-7662-xxxxxxxxxx"},"tx_id":0,"alert":{"action":"allowed","gid":1,"signature_id":4101124,"rev":4,"signature":"SLR Alert - OpenSSL TLS Heartbeat information disclosure (CVE-2014-0160)","category":"Attempted Information Leak","severity":2,"source":{"ip":"10.16.x.x","port":57762},"target":{"ip":"20.x.x.x","port":443},"metadata":{"lock":["false"],"type":["snort","suricata"],"policy":["snort-ips","snort-ids","suricata-ips","suricata-ids"],"created_at":["2
2023-09-25T16:07:57Z IDPS-EVT: 019_01_01","2019_01_01"],"detector_id":["101040"],"severity":["75"],"confidence":["70"],"exploited":["None"],"blacklist_mode":["TEST"],"ids_mode":["REAL"],"threat_name":["CVE-2014-0160 Exploit"],"threat_class_name":["hacking tool"],"server_side":["True"],"flip_endpoints":"serial":"33:00:XX:17:09:XX:XX:5B:F3:3C:XX:E4:72:00:00:00:XX:17:09","fingerprint":"6a:b3:c4:4b:8e:94:ab:49:2a:ae:50:xx:e9:26:3a:xx:xx:xx:xx:xx",","version":"TLS 1.2","notbefore":"2023-05-16T13:31:02","notafter":"2024-05-10T13:31:02","ja3":{"hash":"7c410ce832e848a3321432c9xxxxxx","string":"771,49196-xxxxx-xxxxx-49199-159-158-xxxxx-xxxxx-49192-49191-49162-xxxx-xxxx-49171-57-51-157-156-61-60-53-47-10-106-64-56-5
2023-09-25T16:07:57Z IDPS-EVT: 0-19,0-10-11-13-35-23-65281,29-23-24,0"},"ja3s":{"hash":"ae4exxxxx4d08308082ad26xxxxxx","string":"771,00000,23-65281"}},"app_proto":"tls","flow":{"pkts_toserver":5,"pkts_toclient":6,"bytes_toserver":1157,"bytes_toclient":4751,"start":"2023-09-25T16:07:22.570553+0000"}}
Environment
Cause
- The default syslog message length in the ESXi host is 1024 bytes.
- The vmsyslog daemon is responsible for this truncation on the host for the syslog messages greater than 1024 bytes.
- If we check the nsx-idps-events.log file within the ESXi host, we will see the same truncated logs in the ESXi as well.
Resolution
esxcli system syslog reload
The syslog setting in the ESXi host can be verified using the below command:
- For the changes to take effect, we must ensure that the syslog server configured in the ESXi host is based on a TCP protocol.
- This setting will not affect the UDP-configured syslog server, and the logs will continue to get delivered in a truncated manner to the UDP syslog server. RFC 5426 sets the maximum message transmission length for the UDP protocol to 480 bytes for IPV4 and 1180 bytes for IPV6. Because of this restriction, and because UDP packets can be arbitrarily dropped by the networking infrastructure, the use of UDP for transmitting critical syslog messages is not recommended.
Additional Information
https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-esxi-upgrade/GUID-8981F5FA-BB2A-47FB-A59A-7FC5C523CFDE.html
Impact/Risks:
Due to this default message length present in the ESXi host and IDPS events being larger in size, the events get truncated at the host level and are sent the same way to the syslog server.
Feedback
