Application on NSX node has crashed and core.HeatMap-MsgHand.#####.gz Core dump detected
search cancel

Application on NSX node has crashed and core.HeatMap-MsgHand.#####.gz Core dump detected

book

Article ID: 409880

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • Received an alert on NSX manager indicating that an NSX application has crashed.
    Following  is a sample alarm - 
Infrastructure Service              Application crashed                xxxnsxmgr (xxx.xxx.xxx.11)            Cluster Node Config               Critical            Sep 1, 2025, 8:37:27AM

Description                    Application on NSX node xxx1t1nsxmgr has crashed. The member of core files is 1. Collect the Support bundle including core dump files and contact VMWare support team.

Recommended Action            Collect Support Bundle for NSX node xxxnsxmgr using NSX Manager API. Note. Core dump can be set to move or copy into NSX Tech Support Bundle in order to remove or preserve the local copy on the node. Copy of Support Bundle with core dump files is essential for VMWare Support team to troubleshoot the issue and it is best recommended to save a latest copy of Tech support Bundle including core dump files before removing core dump files from system. Refer KB article for more details.

Reported by Node                  xxxnsxmgr (xxx.xxx.xxx.11)
  • On the NSX manager, you may see core files similar to the following:
    /var/log/core/core.HeatMap-MsgHand.1755844380.3074213.113.6.gz

  • Review the log files below to verify if the system is experiencing the same issue. C1/C2 compiler issue

  • var/log/dumpcore.log file reports the following:

    1 2025-08-22T06:33:00.822Z nsx-mgr-01 NSX 3080367 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="root" level="INFO"] Core dump generation received by process: 3074213 [HeatMap-MsgHand]
    1 2025-08-22T06:33:00.833Z nsx-mgr-01 NSX 3080367 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="root" level="WARNING"] Core file generated: /var/log/core/core.HeatMap-MsgHand.1755844380.3074213.113.6.gz

  • var/log/proton/localhost.log file reports java.lang.NullPointerException

    2025-08-22T06:23:24.336Z SEVERE org.apache.catalina.core.StandardWrapperValve invoke Servlet.service() for servlet [spring] in context with path [/nsxapi] threw exception [Request processing failed; nested exception is java.lang.NullPointerException] with root cause
    java.lang.NullPointerException
            at org.springframework.security.web.header.HeaderWriterFilter$HeaderWriterRequestDispatcher.forward(HeaderWriterFilter.java:170)
            at org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel(InternalResourceView.java:171)
            at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:316)
            at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1406)
            at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1150)
            at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089)
            at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965)
            at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
            at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:529)
            at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:623)

  • Log entries on /var/log/proton/proton-tomcat-wrapper.log.4 file looks similar to the below

    INFO   | jvm 1    | 2025/08/22 06:33:00 | # A fatal error has been detected by the Java Runtime Environment:
    INFO   | jvm 1    | 2025/08/22 06:33:00 | #
    INFO   | jvm 1    | 2025/08/22 06:33:00 | #  SIGSEGV (0xb) at pc=0x000069254d1a4be0, pid=3074213, tid=3078048
    INFO   | jvm 1    | 2025/08/22 06:33:00 | #
    INFO   | jvm 1    | 2025/08/22 06:33:00 | # JRE version: OpenJDK Runtime Environment (11.0.23+10) (build 11.0.23+10-LTS)
    INFO   | jvm 1    | 2025/08/22 06:33:00 | # Java VM: OpenJDK 64-Bit Server VM (11.0.23+10-LTS, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
    INFO   | jvm 1    | 2025/08/22 06:33:00 | # Problematic frame:
    INFO   | jvm 1    | 2025/08/22 06:33:00 | # J 38717 c2

  • /var/log/proton/hs_err_pid3074213.log :

    Current thread (0x00006922e400b000):  JavaThread "HeatMap-MsgHandler-Thread-3" daemon [_thread_in_Java, id=3078048, stack(0x00006921a408a000,0x00006921a418a000)]

    Stack: [0x00006921a408a000,0x00006921a418a000],  sp=0x00006921a4188080,  free space=1016k
    Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
    J 38717 c2 com.vmware.nsx.management.switching.infrastructure.dao.TransportZoneQueryDao$AjcClosure1.run([Ljava/lang/Object;)Ljava/lang/Object; (27 bytes) @ 0x000069254d1a4be0 [0x000069254d1a42e0+0x0000000000000900]
    J 34395 c2 io.micrometer.core.aop.TimedAspect.timedMethod(Lorg/aspectj/lang/ProceedingJoinPoint;)Ljava/lang/Object; (127 bytes) @ 0x000069254b5181c8 [0x000069254b517c00+0x00000000000005c8]
    J 33236 c1 com.vmware.nsx.management.switching.infrastructure.dao.TransportZoneQueryDao.getTransportZone(Lcom/vmware/nsx/management/common/Identifier;)Lcom/vmware/nsx/management/switching/infrastructure/model/TransportZone; (63 bytes) @ 0x00006925477c526c [0x00006925477c4440+0x0000000000000e2c]
    J 33332 c1 com.vmware.nsx.management.switching.infrastructure.service.TransportNodeServiceImpl.getEffectiveTransportProfiles(Lcom/vmware/nsx/management/switching/infrastructure/model/TransportZoneEndpoint;Ljava/util/Map;)Ljava/util/Map; (151 bytes) @ 0x00006925477dfcc4 [0x00006925477df9a0+0x0000000000000324]
    J 34380 c1

Environment

VMware NSX 4.2.0

Cause

The issue is related to OpenJDK 11.0.23, specifically, the C1/C2 compiler, indicating that this is a JRE bug.
In NSX 4.2.0, the JDK version used is openjdk-java11: 11.0.23+10-nn1.

Resolution

  • This is not an issue with NSX.
  • Starting with NSX 4.2.2, we use OpenJDK 17 (version 17.0.14+11-nn1), which includes fixes related to the Java C1/C2 compiler.
  • To resolve the issue, upgrade to NSX 4.2.2 or later.

Additional Information

Reference: 
https://bugs.openjdk.org/browse/JDK-8223613
https://bugs.openjdk.org/browse/JDK-8290864

After collecting the support bundle with core files, the application crash alarm can be resolved by removing the core dump files from the respective NSX nodes.
Note : Please use a file transfer application to save the core-dump file in a safe location.
To remove the core dump file, follow the steps below:
    1. SSH into the NSX Manager using the admin user.
    2. Remove all core-dump files in the core file paths using the below command:
      • nsxcli> del core-dump all
    3. To delete specific core-dump files, run the following command:
      • nsxcli> get core-dump   #This command will list all core-dump files.
      • nsxcli> del core-dump <core-dump-file>   #Specify the name of the core-dump file.
Powered by Wolken Software