Application on NSX node <node> has crashed alarm on NSX Manager due to out of memory on proton service.

Products

• VMware NSX

Issue/Introduction

• NSX Manager reports the "Application has crashed" alarm:
  Application on NSX node <node> has crashed. The number of core files found is 1. Collect the Support Bundle including core dump files and contact VMware Support team. Recommended Action Collect Support Bundle for NSX node <nsx manager> using NSX Manager UI or API.
• CLI command "get core-dumps" shows "proton_oom.hprof" core dump file in Manager's /image/core/.
• nsxapi log file on the impacted NSX Manager shows very high number (usually thousands) of "Start watching file" entries, e.g.:
  /var/log/proton/nsxapi.log
2025-02-25T10:45:12.092Z  INFO l3-tasks1 MultiFileWatcher 4057645 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Start watching file /config/manager/security.properties
2025-02-25T10:45:12.162Z  INFO l3-tasks1 MultiFileWatcher 4057645 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Start watching file /config/manager/security.properties
2025-02-25T10:45:12.187Z  INFO l3-tasks1 MultiFileWatcher 4057645 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Start watching file /config/manager/security.properties

• You may observe an indication of proton being in out of memory state:
  proton/proton-tomcat-wrapper.log
INFO   | jvm 1    | 2025/02/09 13:35:23 | java.lang.OutOfMemoryError: Java heap space
STATUS | wrapper  | 2025/02/09 13:35:23 | The JVM has run out of memory.  Requesting thread dump.
STATUS | wrapper  | 2025/02/09 13:35:23 | Dumping JVM state.
INFO   | jvm 1    | 2025/02/09 13:35:23 | Dumping heap to /image/core/proton_oom.hprof ...
INFO   | jvm 1    | 2025/02/09 13:35:45 | Heap dump file created [4768751237 bytes in 21.607 secs]
INFO   | jvm 1    | 2025/02/09 13:35:57 | #
INFO   | jvm 1    | 2025/02/09 13:35:57 | # java.lang.OutOfMemoryError: Java heap space
STATUS | wrapper  | 2025/02/09 13:35:57 | The JVM has run out of memory.  Requesting thread dump.
STATUS | wrapper  | 2025/02/09 13:35:57 | Dumping JVM state.

• Note this issue can also occur on a Local Manager or a Global Manager in a Federation setup.

 

Environment

VMware NSX release earlier than 4.2.0. 

Cause

This issue is caused by memory exhaustion on NSX Manager's proton service due to multiple attempts to start the watcher process for security.properties configuration file.

Resolution

This issue is resolved in VMware NSX 4.2.0 and higher, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the KB Download Broadcom products and software.

For a workaround, a rolling restart of the proton service, or rolling reboot of NSX Managers can be done:

1. SSH to the first NSX Manager in the cluster as admin.
2. Restart proton service:
   > restart service manager
3. Wait for the service to come up, you can confirm it's status with:
   > get service manager
4. Proceed with remaining Managers in the cluster until the service has been restarted on all nodes. 

Refer to KB article Application on NSX node has crashed alarm for steps on managing the core files themselves and clearing the NSX UI alarm. 

Additional Information

VMware NSX 4.2.0 Release Notes
Fixed Issue 3387947: The proton services crashes due to out of memory.
Out of memory results in core dump. Due to this, you might experience unexpected crush and restart of the proton services.
Do a rolling restart of the proton service on the affected nodes in the cluster.