NSX Manager disconnects from vCenter Server due to the memory leak

Products

• VMware NSX for vSphere

Issue/Introduction

• When running the show process list command in the NSX Manager CLI, you see entries similar to:
  
  %CPU %MEM VSZ RSZ STAT STARTEDTIME COMMAND
  0.0 0.0 4208 1492 Ss Jun 20 00:00:08 init [3]
  0.0 0.0 0 0 S Jun 20 00:00:00 [kthreadd]
  0.0 0.0 105100 3100 Sl Jun 20 00:08:08 /usr/lib/tanuki/bin/./wrapper /usr/lib/tanuki/bin/../conf/appm
   
• In the vsm.log file, you see entries similar to:
  
  # less /home/secureall/secureall/logs/vsm.log
  ess vsm.log.1 | grep -E 'Disconnect default vc connection|Successfully connected to VC'
  2017-02-07 07:37:45.219 GMT INFO TaskFrameworkExecutor-12 DefaultVcConnection:276 - Disconnect default vc connection
  2017-02-07 07:37:46.796 GMT INFO TaskFrameworkExecutor-12 DefaultVcConnection:276 - Disconnect default vc connection
  2017-02-07 07:38:19.511 GMT INFO ViInventoryThread DefaultVcConnection:276 - Disconnect default vc connection
  2017-02-07 07:38:20.158 GMT INFO VcEventsReaderThread DefaultVcConnection:276 - Disconnect default vc connection
  2017-02-07 07:38:24.636 GMT INFO DefaultVcConnectionKeepaliveThread DefaultVcConnection:276 - Disconnect default vc connection
  2017-02-07 07:38:36.021 GMT INFO VCDefaultConnectThread ConnectionHandler:443 - Successfully connected to VC
  2017-02-07 07:38:36.029 GMT INFO ViInventoryThread ConnectionHandler:443 - Successfully connected to VC
  2017-02-07 07:38:41.676 GMT INFO systemEventsPool-1 DefaultVcConnection:276 - Disconnect default vc connection
  2017-02-07 07:38:41.690 GMT INFO systemEventsPool-1 DefaultVcConnection:276 - Disconnect default vc connection
   
• When OOMkiller killed vsm (API server) JVM, tanuki wrapper tries to restart but fails due to insufficient memory.
   
• In the /var/log/nsx-tcserver-wrapper.log file, you see entries similar to:
  
  # less /var/log/nsx-tcserver-wrapper.log
  "FATAL | wrapper | 2017/02/14 00:40:09 | Could not spawn JVM process: Cannot allocate memory".
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX for vSphere 6.2.x
VMware NSX for vSphere 6.3.x

Cause

This issue occur because logging process of tanuki watchdog application consumes a lot of memory causing the memory leak.

Resolution

This issue is resolved in NSX for vSphere 6.3.0

 

To work around this issue if you do not want to upgrade, file a support request with Broadcom support and quote this Knowledge Base article ID (318595) in the problem description.

Additional Information