Memory of "sha" service on the NSX-T Unified Appliance keeps increasing due to a memory leak.

Products

• VMware NSX

Issue/Introduction

• Alarm "The memory usage on Manager node <node-uuid> has reached <system_usage>% which is at or above the high threshold value of <threshold>%."
• Memory usage on NSX Manager node(s) keeps increasing, hitting 90% threshold and beyond. 
• In /var/log/vmware/top-mem.log on the NSX Manager, the memory consumption of sha is observed to be steadily increasing:
  
  Day 1:
  

  PID  USER    PR NI    VIRT    RES  SHR S  %CPU  %MEM   TIME+ TGID COMMAND
  1367 nsx-sha 20  0 4063404 358076 6860 S   0.0   1.5 4326:43 1367 /opt/vmware/nsx-netopa/libexec/python-3/bin/python3 /opt/vmware/nsx-netopa/bin/agent.py

  
  
  Day 2:
  

  1367 nsx-sha 20  0 4063404 384184 6832 S   0.0   1.6 4753:53 1367 /opt/vmware/nsx-netopa/libexec/python-3/bin/python3 /opt/vmware/nsx-netopa/bin/agent.py

  
  
  Day 3:
  

  1367 nsx-sha 20  0 4063404 409564 6880 S   0.0   1.7 5185:39 1367 /opt/vmware/nsx-netopa/libexec/python-3/bin/python3 /opt/vmware/nsx-netopa/bin/agent.py

   

  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

• NSX-T Data Center 3.2.x
• VMware NSX versions earlier than 4.1.2.5.

Cause

The NSX Manager's sha service has a memory leak which is causing increasingly high memory consumption and will lead to this alarm.

Resolution

This issue is resolved in VMware NSX 4.1.2.5 and 4.2.1 available at Broadcom Downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

 

Workaround:
Restarting SHA on NSX Manager will release the leaked memory:

1. SSH to the impacted NSX Manager as root.
2. Restart the service:
   /etc/init.d/nsx-sha restart

Notes:

1. This workaround will not impact any functionality on NSX Manager, and it will not have any effect on the datapath. 
2. The workaround will temporarily resolve memory high memory issue (memory of the new SHA process will increase again), hence the workaround will need to be repeated.