VMware NSX 4.1 sudo or nvpapi application on NSX node has crashed

Products

• VMware NSX

Issue/Introduction

• Application crash alarms are triggered for an NSX component.
• There is minimal noticeable production impact.
• In /var/log/syslog of the NSX manager you can confirm which NSX component (Manager or Edge) has the error (in this case edge##):

  NSX 121272 - [nsx@6876 comp="nsx-manager" subcomp="nsx-sha" username="nsx-sha" level="CRITICAL" eventFeatureName="infrastructure_service" eventType="application_crashed" eventSev="critical" eventState="On" entId="cd59fec3-####-####-####-##########6d"] Application on NSX node <edge_name> has crashed. 
  
  The number of core files found is 1. Collect the Support Bundle including core dump files and contact VMware Support team.

• Checking /var/log/dumpcore.log of the above mentioned NSX node we see a core file has been generated for sudo, nvpapi, or api_roothelper:

  NSX 4187495 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="root" level="INFO"] Core dump generation received by process: None [nvpapi]
  
  NSX 4187495 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="root" level="WARNING"] Core file generated: /var/log/core/core.nvpapi.py.############.gz

• This can be further confirmed checking /var/log/syslog of the above node for core dumps:

  edge##.#######.#### NSX 1366408 - [nsx@6876 comp="nsx-edge" subcomp="node-mgmt" username="root" level="WARNING"] Core file generated: /var/log/core/core.nvpapi.py.###############.gz
  
  or
  
  edge##.#######.#### NSX 1366408 - [nsx@6876 comp="nsx-edge" subcomp="node-mgmt" username="root" level="WARNING"] Core file generated: /var/log/core/core.sudo.##############.gz

 

• Checking in /var/log/core/ on the node you see the core files:

  root@edge##:/var/log/core# ls
  
  total 454M
  
  -rw-rw-rw- 1 root root 321M Jun 26 12:39 core.nvpapi.py.###############.gz

• In /var/log/kern.log you will see a line similar to the line below:

  <manager_name> kernel - - - [17318523.570124] api_roothelper.[1097264] bad frame in rt_sigreturn frame:000##########1b8 ip:72a######30b sp:757######ec0 orax:fff##########fff in libc-2.31.so[72a#########+####00]
  
  <manager_name> kernel - - - [17318523.570144] grsec: Segmentation fault occurred at 0000000000000000 in /opt/vmware/nsx-node-api/bin/api_roothelper.sh[api_roothelper.:1097264] uid/euid:0/0 gid/egid:0/0, parent /usr/bin/sudo[sudo:1097263] uid/euid:0/0 gid/egid:0/0

NOTE: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware NSX

Cause

This is a known issue caused by the kernel version of the base appliance in NSX 4.1.0.0 

 

Services have crashed and the system generated their respective cored dump files. All NSX services are configured to be auto-restarted after hitting a crash. Depending on the application which has crashed it might be possible other services depending on it may not be functioning correctly.

Resolution

This issue is resolved in VMware NSX 4.1.1, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.


Workaround

• It is recommended to verify the services status which have crashed to confirm whether it's running. On the NSX-T appliance node, service status can be verified in nsxcli as below:

  nsxcli> get service <service-name>
  
  or
  
  nsxcli> get services 

• After collecting the support-bundle, the application crashed alarm can be resolved by removing the core dump files from the respective nodes.

• • On NSX appliance nodes, the following nsxcli command can be used to remove core and heap dump files:

nsxcli> del core-dump all
or
nsxcli> del core-dump <core-dump-file>

 

• • On ESXi host transport nodes, the following command can be used when logged in from the host as root:

root> rm -rf /var/core/*

 

 

Additional Information

Application on NSX node has crashed alarm