Memory of SHA on UA keeps increasing due to memory leak.

Products

• VMware NSX

Issue/Introduction

• Alarm "The memory usage on Manager node <node-uuid> has reached <system_usage>% which is at or above the high threshold value of <threshold>%."
• Memory usage on NSX Manager node(s) keeps increasing, hitting 90% threshold and beyond. 
• In NSX Manager's logs, memory of nsx-sha is steadily increasing, e.g.:
  /var/log/vmware/top-mem.log:
  day 1:
  PID  USER    PR NI    VIRT    RES  SHR S  %CPU  %MEM   TIME+ TGID COMMAND
1367 nsx-sha 20  0 4063404 358076 6860 S   0.0   1.5 4326:43 1367 /opt/vmware/nsx-netopa/libexec/python-3/bin/python3 /opt/vmware/nsx-netopa/bin/agent.py
  day 2:
  1367 nsx-sha 20  0 4063404 384184 6832 S   0.0   1.6 4753:53 1367 /opt/vmware/nsx-netopa/libexec/python-3/bin/python3 /opt/vmware/nsx-netopa/bin/agent.py
  day 3:
  1367 nsx-sha 20  0 4063404 409564 6880 S   0.0   1.7 5185:39 1367 /opt/vmware/nsx-netopa/libexec/python-3/bin/python3 /opt/vmware/nsx-netopa/bin/agent.py

Environment

• NSX-T Data Center 3.2.x
• NSX earlier than version 4.1.2.5.

Cause

NSX Manager's SHA service has a memory leak which is causing increasingly high memory consumption and will lead to this alarm.

Resolution

This issue is resolved in NSX 4.1.2.5, and in 4.2.1 and higher.

Workaround:
Restarting SHA on NSX Manager will release the leaked memory:

1. SSH to the impacted NSX Manager as root.
2. Restart the service:
   /etc/init.d/nsx-sha restart

Notes:

1. This workaround will not impact any functionality on NSX Manager, and it will not have any effect on the datapath. 
2. The workaround will temporarily resolve memory high memory issue (memory of the new SHA process will increase again), hence the workaround will need to be repeated.