NSX Manager node reports alarm that application has crashed, core dump file: proxy

Products

VMware NSX

Issue/Introduction

Alarms triggered in the NSX UI showing:
"Application on NSX node <node> has crashed. The number of core files found is 1. Collect the Support Bundle including core dump files and contact VMware Support team. Recommended Action Collect Support Bundle for NSX node <nsx manager> using NSX Manager UI or API."
On the manager node the alarm is referring to, a core dump file is found in /image/core: proxy_oom.hprof
In the NSX manager /var/log/proxy/reverse-proxy.log we see the following ERROR:

ERROR GmleServiceClient:worker-0 NettyInboundHandler 83518 - [nsx@6876 comp="nsx-manager" errorCode="MP101" level="ERROR" subcomp="http"] Closing connection NettyConnection(NettyChannel(local=127.#.#.1:40470, remote=127.#.#.1:9823), active=true) because of unhandled exception java.lang.OutOfMemoryError: GC overhead limit exceeded
        at com.vmware.nsx.rpc.transport.netty.NsxRpcMessageDecoder.decodeVersion2(NsxRpcMessageDecoder.java:116)
        at com.vmware.nsx.rpc.transport.netty.NsxRpcMessageDecoder.decode(NsxRpcMessageDecoder.java:90)
        at com.vmware.nsx.rpc.transport.netty.NsxRpcMessageDecoder.decode(NsxRpcMessageDecoder.java:56)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)

The proxy-tomcat-wrapper.log may also report Java heap space and out of memory issues.

INFO   | jvm 1    |  18:27:02 | java.lang.OutOfMemoryError: Java heap space
STATUS | wrapper  |  18:27:02 | The JVM has run out of memory.  Requesting thread dump.
STATUS | wrapper  |  18:27:02 | Dumping JVM state.
STATUS | wrapper  |  18:27:02 | The JVM has run out of memory.  Restarting JVM.
INFO   | jvm 1    |  18:27:02 | Dumping heap to /image/core/proxy_oom.hprof ...
INFO   | jvm 1    |  18:27:02 | 2024-02-25 01:27:02
INFO   | jvm 1    |  18:27:02 | Full thread dump OpenJDK 64-Bit Server VM (25.372-b07 mixed mode):

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX

Cause

This proxy service can crash when security scans are run against the NSX Manager, if they utilize invalid URL's, this can lead an accumulation of session objects in the authentication server and result in the process crashing and restarting, due to out of memory.

Signs of a scanner may be seen in the NSX manager log /var/log/proxy/envoy_access_log.txt or /var/log/proxy/localhost_access.log, depending on the version of NSX deployed.

10.##.#.## 10.##.#.## "GET" "/api/ldap/config/" "HTTP/1.1" 403 UAEX 0 141 1 - "10.##.#.##" "<Scanning tool>" "######-556c-4a4f-9d97-#########" "10.##.#.##" "-"

This log shows the inbound HTTP connections to the NSX manager, the text <Scanning tool> may show the name of the scanner running against the manager, the first entry on the line above 10.##.#.## is the IP address of the device making the HTTP call to the manager.

Resolution

This issue is resolved in VMware NSX 4.2.1, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:

To workaround this issue, exclude the NSX Managers from the security scans and remove the core dump, once the core dump is removed, the alarm will resolved after a few minutes. The services will auto restart after a crash occurs.

Refer to Application on NSX node has crashed alarm for steps on managing the core files themselves and clearing the NSX UI alarm.

Additional Information

If you are contacting Broadcom support about this issue, please provide the following:

NSX Manager support bundles
Text of any error messages seen in NSX GUI or command lines pertinent to the investigation

Handling Log Bundles for offline review with Broadcom support

If the steps here have not resolved the issue for you, you can refer to the following KB which can provide further troubleshooting steps:
Troubleshooting NSX issues