Application on NSX Manager nodes has crashed, core dump file: proxy_oom.hprof is present in /image/core. (CrlWebFetcher.java:49)
search cancel

Application on NSX Manager nodes has crashed, core dump file: proxy_oom.hprof is present in /image/core. (CrlWebFetcher.java:49)

book

Article ID: 392314

calendar_today

Updated On:

Products

VMware NSX VMware NSX-T Data Center

Issue/Introduction

  • Alarm feature: infrastructure_service
  • JVM has run out of memory
  • Crashed producing proxy_oom.hprof core dump
    /var/log/proxy/proxy-tomcat-wrapper.log 
    INFO   | jvm 1    | 2024/10/10 19:23:19 | # java.lang.OutOfMemoryError: Java heap space
    STATUS | wrapper  | 2024/10/10 19:23:19 | The JVM has run out of memory.  Requesting thread dump.
    STATUS | wrapper  | 2024/10/10 19:23:19 | Dumping JVM state.
    STATUS | wrapper  | 2024/10/10 19:23:19 | The JVM has run out of memory.  Restart JVM (Ignoring, already restarting).
    INFO   | jvm 1    | 2024/10/10 19:23:19 | # -XX:OnOutOfMemoryError="/sbin/dumpcore.py tanuki /image/core/proxy_oom.hprof /run/proxy/proxy-jvm.pid"
    INFO   | jvm 1    | 2024/10/10 19:23:19 | #   Executing /bin/sh -c "/sbin/dumpcore.py tanuki /image/core/proxy_oom.hprof /run/proxy/proxy-jvm.pid"...

    INFO   | jvm 1    | 2024/10/10 19:23:44 |       at org.bouncycastle.asn1.ASN1StreamParser.readVector(Unknown Source)
    INFO   | jvm 1    | 2024/10/10 19:23:44 |       at org.bouncycastle.asn1.DERSequenceParser.getLoadedObject(Unknown Source)
    INFO   | jvm 1    | 2024/10/10 19:23:44 |       at org.bouncycastle.asn1.DERSequenceParser.toASN1Primitive(Unknown Source)
    INFO   | jvm 1    | 2024/10/10 19:23:44 |       at org.bouncycastle.jcajce.provider.CertificateFactory.readDERCRL(Unknown Source)
    INFO   | jvm 1    | 2024/10/10 19:23:44 |       at org.bouncycastle.jcajce.provider.CertificateFactory.readCrl(Unknown Source)
    INFO   | jvm 1    | 2024/10/10 19:23:44 |       at org.bouncycastle.jcajce.provider.CertificateFactory.engineGenerateCRL(Unknown Source)
    INFO   | jvm 1    | 2024/10/10 19:23:44 |       at java.security.cert.CertificateFactory.generateCRL(CertificateFactory.java:497)
    INFO   | jvm 1    | 2024/10/10 19:23:44 |       at com.vmware.nsx.management.security.CrlFetcher.readCrlFromStream(CrlFetcher.java:33)
    INFO   | jvm 1    | 2024/10/10 19:23:44 |       at com.vmware.nsx.management.security.CrlWebDirectFetcher.downloadCrlFromWeb(CrlWebDirectFetcher.java:148)
    INFO   | jvm 1    | 2024/10/10 19:23:44 |       at com.vmware.nsx.management.security.CrlWebDirectFetcher.downloadCrl(CrlWebDirectFetcher.java:44)
    INFO   | jvm 1    | 2024/10/10 19:23:44 |       at com.vmware.nsx.management.security.CrlWebDirectFetcher.fetch(CrlWebDirectFetcher.java:35)
    INFO   | jvm 1    | 2024/10/10 19:23:44 |       at com.vmware.nsx.management.security.CrlWebFetcher.fetch(CrlWebFetcher.java:49)
    INFO   | jvm 1    | 2024/10/10 19:23:44 |       at com.vmware.nsx.management.security.CdpCrlChecker.checkRevocation(CdpCrlChecker.java:160)
    INFO   | jvm 1    | 2024/10/10 19:23:44 |       at com.vmware.nsx.management.security.CdpCrlChecker.checkRevocation(CdpCrlChecker.java:130)
    INFO   | jvm 1    | 2024/10/10 19:23:44 |       at com.vmware.nsx.management.security.NsxTrustManager.checkCertificateValid(NsxTrustManager.java:346)
    INFO   | jvm 1    | 2024/10/10 19:23:44 |       at com.vmware.nsx.management.security.NsxTrustManager._checkServerTrusted(NsxTrustManager.java:305)
    INFO   | jvm 1    | 2024/10/10 19:23:44 |       at com.vmware.nsx.management.security.NsxTrustManager.checkServerTrusted(NsxTrustManager.java:271)

    ERROR  | wrapper  | 2024/10/10 19:23:47 | Shutdown failed: Timed out waiting for signal from JVM.
    STATUS | wrapper  | 2024/10/10 19:23:47 | Dumping JVM state.
    ERROR  | wrapper  | 2024/10/10 19:23:52 | JVM did not exit on request, termination requested.
    STATUS | wrapper  | 2024/10/10 19:23:52 | JVM received a signal SIGKILL (9).
    STATUS | wrapper  | 2024/10/10 19:23:52 | JVM process is gone.
    STATUS | wrapper  | 2024/10/10 19:23:52 | JVM exited after being requested to terminate.
    STATUS | wrapper  | 2024/10/10 19:24:08 | JVM process is gone.


  • Bundle logs related from NSX Manager:
    /var/log/syslog
    nsxtctrl NSX 1421 - [nsx@6876 comp="nsx-manager" subcomp="ip_utils" username="nsx-sha" level="INFO"] Delivered alarm event:AlarmMsg: feature:infrastructure_service,       
    status:{9: {UUID(): (True, {'node_display_or_host_name': 'nsxtctrl1', 'core_dump_count': '1'})}}, event_type:application_crashed, event_type_id:None,      
    deleted_entity_list:None to alarm framework successfully.

Environment

VMware NSX 4.1.1

Cause

The proxy is performing a CRL check on the certificates, and the CRL for one or more certificates is likely exceptionally large. As it is loaded into the proxy's memory, it's causing the proxy to exceed its memory capacity.

Resolution

Workaround:

Disable CRL checking with the following API:

curl -u admin:${PASSWORD} -i -k -X PUT https://$IP/api/v1/global-configs/SecurityGlobalConfig -H "Content-Type:application/json" -d '{ "_revision" : 0, "resource_type": "SecurityGlobalConfig", "crl_checking_enabled" : "false", "ca_signed_only" : "false" }'

 

Refer to Application on NSX node has crashed alarm for steps on managing the core files themselves and clearing the NSX UI alarm. 

Additional Information

If you are contacting Broadcom support about this issue, please provide the following:

  • NSX Manager support bundles
  • Text of any error messages seen in NSX GUI or command lines pertinent to the investigation

Handling Log Bundles for offline review with Broadcom support

If the steps here have not resolved the issue for you, you can refer to the following KB which can provide further troubleshooting steps:

Troubleshooting NSX issues

Powered by Wolken Software