nsx-exporter process crash on ESXi host causes an NSX alarm, Application on NSX node <hostname> has crashed

Products

• VMware NSX

Issue/Introduction

• Alarms reported for application crashes on ESXi transport nodes.
• Logs similar to the below observed on the ESXi host in /var/run/log/vobd.log

2024-04-02T18:52:43.225Z In(14) vobd[2098026]:  [UserWorldCorrelator] 213546302089us: [vob.uw.core.dumped] /usr/lib64/vmware/nsx-exporter/nsx-exporter(2101503) /var/core/nsx-exporter-zdump.000
2024-04-02T18:52:43.225Z In(14) vobd[2098026]:  [UserWorldCorrelator] 213543013185us: [esx.problem.application.core.dumped] An application (/usr/lib64/vmware/nsx-exporter/nsx-exporter) running on ESXi host has crashed (1 time(s) so far). A core file may have been created at /var/core/nsx-exporter-zdump.000.
 

• Logs similar to the below will be observed on the ESXi host in /var/run/log/vmkernel.log

2024-04-02T18:52:41.442Z In(182) vmkernel: cpu76:2101643)User: 3238: nsx-exporter: wantCoreDump:nsx-exporter signal:11 exitCode:0 coredump:enabled
2024-04-02T18:52:41.545Z In(182) vmkernel: cpu76:2101643)UserDump: 3072: nsx-exporter: Dumping cartel 2101503 (from world 2101643) to file /var/core/nsx-exporter-zdump.000 ...
2024-04-02T18:52:43.225Z In(182) vmkernel: cpu67:2101643)UserDump: 3367: nsx-exporter: Userworld(nsx-exporter) coredump complete.

 

• Logs similar to the below indicating that the flow count exceeded the buffer will be observed in  /var/run/log/nsx-syslog

2024-04-02T18:52:41.373Z In(182) nsx-exporter[2101503]: NSX 2101503 - [nsx@6876 comp="nsx-esx" subcomp="agg-service" >tid="2101643" level="INFO"] Flow count diffs in actual (27368) vs in FLOW_GET_NUMRECORDS (27367)

 

Note: In the case of any core dump on an ESXi host (NSX transport node), the first three points should always be there, point two and three will differ in that the name of the core dump will match the service which crashed. The matching entry here is the fourth point, with the log entry.

Environment

VMware NSX 4.x

Cause

This issue occurs when the flow count exceeds the buffer.

Resolution

This issue is resolved in VMware NSX 4.1.2.5
This issue is resolved in VMware NSX 4.2.0

Additional Information

Impact/Risks:
This issue is very rare, and when the issue occurs the application will restart, so it is unlikely that much impact will be observed.