Datapath crashed after new Edge VM was deployed

Products

• VMware NSX

Issue/Introduction

• Deployed new NSX Edge VM.
• Message at /var/log/syslog. For details refer to Application on NSX node has crashed alarm
  
  Application on NSX node <NODE_NAME> has crashed. The number of core files found is 1. Collect the Support Bundle including core dump files and contact VMware Support team.
  
  
• Device activation fails at /var/log/syslog
  
  <TIMESTAMP> <HOSTNAME> NSX 4912 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc##" level="WARN"] vmxnet3_dev_start(): Device activation: UNSUCCESSFUL
<TIMESTAMP> <HOSTNAME> NSX 4912 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc##" level="WARN"] Port 1 start error: Invalid argument
<TIMESTAMP> <HOSTNAME> NSX 4912 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc##" level="WARN"] Failed to init port 1 : Invalid argument
<TIMESTAMP> <HOSTNAME> NSX 4912 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="phys-port" tname="dp-ipc##" level="WARN"] Failed to start port 1 for MTU change (mtu: 9000) ret: -22; attempting to roll back config
...
<TIMESTAMP> <HOSTNAME> datapath-systemd-helper 4768 - -  <TIMESTAMP> datapathd 4912 intel-rte tname="dp-ipc##" [FATAL] PANIC in phys_port_set_mtu():
<TIMESTAMP> <HOSTNAME> datapath-systemd-helper 4768 - -  <TIMESTAMP> datapathd 4912 intel-rte tname="dp-ipc##" [FATAL] Rollback failed. Bailing out.
<TIMESTAMP> <HOSTNAME> 
<TIMESTAMP> <HOSTNAME> NSX 6395 - [nsx@6876 comp="nsx-edge" subcomp="node-mgmt" username="root" level="WARNING"] Core file generated: /var/log/core/core.dp-ipc##.##########.####.#.#.gz


• ESXi Memory allocation fails at /var/log/vmkernel.log
  
  <TIMESTAMP> In(182) vmkernel: cpu###:<NUMBER>)Vmxnet3: 14590: indLROPktToGuest: 1, vcd->umkShared->vrrsSelected: 8 port <PORT_ID>
<TIMESTAMP> In(182) vmkernel: cpu###:<NUMBER>)Admission failure in path: host/user/vm.<VM_ID>:uwWorldStore.<VM_ID>
<TIMESTAMP> In(182) vmkernel: cpu###:<NUMBER>)uwWorldStore.<VM_ID> (<NUMBER>) requires 4 KB, asked 4 KB from vm.<VM_ID> (<NUMBER>) which has 67579904 KB occupied and 0 KB available.
<TIMESTAMP> In(182) vmkernel: cpu###:<NUMBER>)Admission failure in path: host/user/vm.<VM_ID>:uwWorldStore.<VM_ID>
<TIMESTAMP> In(182) vmkernel: cpu###:<NUMBER>)uwWorldStore.<VM_ID> (<NUMBER>) requires 4 KB, asked 4 KB from vm.<VM_ID> (<NUMBER>) which has 67579904 KB occupied and 0 KB available.
<TIMESTAMP> Wa(180) vmkwarning: cpu###:<NUMBER>)WARNING: World: 2874: Could not allocate new world handle for world ID: 12121398: Admission check failed for memory resource
• vmx file has sched.mem.shareCosBufSize configured. 
  shareCosBufSize is automatically configured to 32 on NSX Edge 4.1 or above.
  $ grep shareCosBufSize edge.vmx
  sched.mem.shareCosBufSize = "32"

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

ESXi 8.0 Update 2
NSX 4.x

Cause

ESXi reports memory space required to vCenter so that DRS can calculate VM placement.
Since ESXi 8.0 Update 2 does not include memory space for vmxnet3, NSX Edge VM may fail to activate vmnic3 and let datapathd restart.

Resolution

Update to VMware ESXi 8.0 Update 3 or later.

To workaround this issue, set advanced DRS options. 

1. Log into vCenter as an administrator using the vSphere Client.
2. Right-click your cluster from the inventory.
3. Click Edit Settings.
4. Click VMware DRS (if it is not enabled, enable it).
5. Click Advanced Options.
6. Add MemOverheadGrowth with a value of 20.
7. Click OK to close the Advanced Options.
8. Click OK to close the cluster configuration.

If you applied this workaround, remove before upgrade to ESXi 8.0 Update 3.