NSX Manager node reports alarm that application has crashed. A proxy_oom.hprof core file is present in /image/core.

Products

• VMware NSX

Issue/Introduction

• Alarms reporting "Application on NSX node <node> has crashed. The number of core files found is 1. Collect the Support Bundle including core dump files and contact VMware Support team. Recommended Action Collect Support Bundle for NSX node <nsx manager> using NSX Manager UI or API." are seen in the NSX UI. When investigating further proxy_oom.hprof core file is found in /image/core on the NSX Manager file system. 
• The reverse-proxy.log may report a Java 'OutOfMemoryError' occurring.

2024-02-25T01:27:07.550Z ERROR GmleServiceClient:worker-0 NettyInboundHandler 83518 - [nsx@6876 comp="nsx-manager" errorCode="MP101" level="ERROR" subcomp="http"] Closing connection NettyConnection(NettyChannel(local=127.0.0.1:40470, remote=127.0.0.1:9823), active=true) because of unhandled exception java.lang.OutOfMemoryError: GC overhead limit exceeded
        at com.vmware.nsx.rpc.transport.netty.NsxRpcMessageDecoder.decodeVersion2(NsxRpcMessageDecoder.java:116)
        at com.vmware.nsx.rpc.transport.netty.NsxRpcMessageDecoder.decode(NsxRpcMessageDecoder.java:90)
        at com.vmware.nsx.rpc.transport.netty.NsxRpcMessageDecoder.decode(NsxRpcMessageDecoder.java:56)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.lang.Thread.run(Thread.java:750)

• The proxy-tomcat-wrapper.log may also report Java heap space and out of memory issues. 

INFO   | jvm 1    | 2024/02/24 18:27:02 | java.lang.OutOfMemoryError: Java heap space
STATUS | wrapper  | 2024/02/24 18:27:02 | The JVM has run out of memory.  Requesting thread dump.
STATUS | wrapper  | 2024/02/24 18:27:02 | Dumping JVM state.
STATUS | wrapper  | 2024/02/24 18:27:02 | The JVM has run out of memory.  Restarting JVM.
INFO   | jvm 1    | 2024/02/24 18:27:02 | Dumping heap to /image/core/proxy_oom.hprof ...
INFO   | jvm 1    | 2024/02/24 18:27:02 | 2024-02-25 01:27:02
INFO   | jvm 1    | 2024/02/24 18:27:02 | Full thread dump OpenJDK 64-Bit Server VM (25.372-b07 mixed mode):

 

 

Cause

This proxy service can crash when security scans are run against the NSX Manager if they utilize invalid URL's which may cause an accumulation of session objects in the authentication server and lead to this process crashing and restarting..

Resolution

To workaround this, exempt the NSX Managers from the security scan. 

 

Additional Information

Refer to Application on NSX node has crashed alarm for steps on managing the core files themselves and clearing the NSX UI alarm.