After an upgrade to NSX 4.x, the alarm Application on NSX node <nsx manager> has crashed is generated

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• You recently completed a VMware NSX upgrade to 4.x.
• You are getting alarms similar to the following: 

Application on NSX node <nsx manager> has crashed. The number of core files found is 1. Collect the Support Bundle including core dump files and contact VMware Support team. Recommended Action Collect Support Bundle for NSX node <nsx manager> using NSX Manager UI or API.

• On the NSX-T manager when ls -l on /image/core/ as root user the following files are seen:

migration_oom.hprof.gz

• The following log entries are seen in the NSX-T manager /var/log/upgrade-coordinator/logical-migration.log:

ERROR netty-0 CorfuRuntime 5668 handleUncaughtThread: netty-0 terminated with throwable of type OutOfMemoryError
java.lang.OutOfMemoryError: Java heap space
ERROR main Migration 5668 - [nsx@6876 comp="nsx-manager" errorCode="MP217" level="ERROR" subcomp="manager"] Migration failed
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space

Environment

VMware NSX-T Data Center

Cause

During the upgrade, the system encountered a memory issue and generated a core dump.

Resolution

If the upgrade is complete, the alarm is still present and the core dump can be removed. See workaround below for steps to remove the core dump.

Workaround:
1. Remove the file in the core file paths as below, as admin user: 

• nsxcli> del core-dump all

or

• nsxcli> del core-dump <core-dump-file>