After an upgrade to NSX 4.x, the alarm Application on NSX node <nsx manager> has crashed is generated

Products

• VMware NSX

Issue/Introduction

• You recently completed a VMware NSX upgrade to 4.x.
• You are getting alarms similar to the following: 

Application on NSX node <nsx manager> has crashed. The number of core files found is 1. Collect the Support Bundle including core dump files and contact VMware Support team. Recommended Action Collect Support Bundle for NSX node <nsx manager> using NSX Manager UI or API.

• On the NSX manager, as root user, you may see files similar to the following:

/image/core/migration_oom.hprof.gz
/var/log/core/core.HeatMap-MsgHand.1727831662.75883.113.6.gz

• You see messages similar to the following in /var/log/upgrade-coordinator/logical-migration.log:

ERROR netty-0 CorfuRuntime 5668 handleUncaughtThread: netty-0 terminated with throwable of type OutOfMemoryError
java.lang.OutOfMemoryError: Java heap space
ERROR main Migration 5668 - [nsx@6876 comp="nsx-manager" errorCode="MP217" level="ERROR" subcomp="manager"] Migration failed
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space

• You see messages similar to the following in /var/log/dumpcore.log:
  
  2024-10-02T01:14:23.077Z ####### NSX 85268 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="root" level="INFO"] Core dump generation received by process: 75883 [HeatMap-MsgHand]
2024-10-02T01:14:23.078Z ####### NSX 85268 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="root" level="WARNING"] Core file generated: /var/log/core/core.HeatMap-MsgHand.1727831662.75883.113.6.gz
  
  

Note: The timing of the messages noted in /var/log/upgrade-coordinator/logical-migration.log and /var/log/dumpcore.log are within the timeframe of the upgrade to 4.x happening.

Environment

VMware NSX

Cause

During the upgrade, the system encountered a memory issue and generated a core dump.

Resolution

If the upgrade is complete, the alarm is still present and the core dump can be removed. See workaround below for steps to remove the core dump.

Workaround:

1. ssh to the nsx manager using the admin user.
2. Remove the all core dump files in the core file paths use below command.
   
   nsxcli> del core-dump all

or

 

3. To delete specific core-dump file, run below command.

• nsxcli> get core-dump # This command will list all core-dump files.
• nsxcli> del core-dump <core-dump-file> # specify the name of the core-dump.