ESXi host applications localcli, esxcfg-vmknic, and python3 crash due to nsx-exporter Out of Memory (OOM) when multiple DVSes are configured

Products

• VMware NSX

Issue/Introduction

• Multiple ESXi host applications (localcli, esxcfg-vmknic, python3, memstats, nsx-exporter) crash and generate encrypted core dumps.

• The /var/run/log/vobd.log on the ESXi host reports repeated crash events similar to the following:

  [esx.problem.application.core.dumped.encrypted] An application (/bin/esxcfg-vmknic) running on ESXi host has crashed (1198 time(s) so far).
  [esx.problem.application.core.dumped.encrypted] An application (/bin/localcli) running on ESXi host has crashed (8847 time(s) so far).
  

   

• Log strings indicate admission failures for the nsx-exporter pool:

  Admission failure in path: host/vim/vmvisor/nsx-exporter
  

 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX

VMware NSX-T Data Center

Cause

 A memory leak exists in the nsx-exporter service when an ESXi host is connected to multiple vSphere Distributed Switches (vDS) and an external application (such as VMware Aria Automation or Aria Operations for Networks) frequently polls the GetTunnelsStatus API (/api/v1/transport-nodes/<node-id>/tunnels). During this API call, memory is only freed for the last DVS in the array, causing the memory for the remaining DVS to leak and eventually exhaust the nsx-exporter resource pool limit.

Resolution

This issue is resolved in VMware NSX 4.2.0, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

 

Workaround: If you believe you have encountered this issue and are unable to upgrade, open a support case with Broadcom Support and refer to this KB article.

To temporarily mitigate the issue:

1. Restart the nsx-exporter service on the affected ESXi host to clear the leaked memory: /etc/init.d/nsx-exporter restart.

2. If feasible, consolidate the ESXi host networking to use a single DVS, as the memory leak does not occur in single-DVS configurations.

3. Reduce the polling frequency of external monitoring or automation tools querying the NSX Manager tunnels API.

 

Additional Information

If you are contacting Broadcom support about this issue, please provide the following:

• NSX Manager support bundles
• ESXi host support bundles
• The text of any error messages seen in the NSX GUI or command lines pertinent to the investigation.