PSOD on the ESXi host during host preparation for NSX-T with "ip_init: failed to allocate flowtable"

Products

• VMware NSX
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

ESXi host is being prepared for NSX-T.

Workloads (Virtual Machines) are running on the host during host preparation.

ESXi host may experience a Purple Screen of Death (PSOD) with backtrace similar to:
ip_init: failed to allocate flowtable
Code start: 0x42000f400000 VMK uptime: 87:01:14:56.283
0x4539b829b180:[0x42000f4ff0a3]PanicvPanicInt@vmkernel#nover+0x327 stack: 0x4539b829b288
0x4539b829b250:[0x42000f4ff90a]Panic_vPanic@vmkernel#nover+0x23 stack: 0x41ffd3814298
0x4539b829b270:[0x42000f5165aa]vmk_vPanic@vmkernel#nover+0xf stack: 0x8000
0x4539b829b280:[0x42001076ff3b]panic@(tcpip4)#<None>+0x4c stack: 0x4539b829b2e0
0x4539b829b2e0:[0x4200107c3669]ip_init@(tcpip4)#<None>+0x686 stack: 0x41ffd3802f60
0x4539b829b320:[0x42001078d5da]domain_init@(tcpip4)#<None>+0x2f stack: 0x431a52901080
0x4539b829b340:[0x42001077462e]vmk_net_instance_init@(tcpip4)#<None>+0x463 stack: 0x2af800000080
0x4539b829b3f0:[0x4200107652f6]Tcpip_InitializeStackInstance@(tcpip4)#<None>+0x217 stack: 0x2
0x4539b829b4b0:[0x42000f6be9df]Net_TcpipCreateStackInstance@vmkernel#nover+0xe4 stack: 0x3
0x4539b829b500:[0x42000f706e53]NetVsi_TcpipCreateNetinstanceSet@vmkernel#nover+0x184 stack: 0x3b
0x4539b829b5a0:[0x42000f4023dd]VSI_SetInfo@vmkernel#nover+0x2ca stack: 0x4539b829b6a0
0x4539b829b620:[0x42000f92a702]UW64VMKSyscallUnpackVSI_Set@vmkernel#nover+0x1eb stack: 0x4539b829b8e0
0x4539b829bee0:[0x42000f8b408a]User_UWVMK64SyscallHandler@vmkernel#nover+0x183 stack: 0xd783bed685fd845
0x4539b829bf40:[0x42000f54b4b8]SyscallUWVMK64@vmkernel#nover+0x90 stack: 0x0

Environment

VMware vSphere ESXi 7.0
VMware NSX-T

Cause

The issue is from not having enough TCP/IP heap space. If the ESXi host is running low on memory. It can't allocate the heap space needed to become an NSX Host Transport Node. This can lead the kernel to exhaust its memory completely. That results in the ESXi host crashing with a Purple Screen of Death (PSOD).

Resolution

The issue with ESXi TCP/IP heap is fixed in 7.0 u3o.

Workaround:

Reduce memory consumption on the ESXi host (e.g. by managing number of VMs on the host, or putting the host to Maintenance Mode).

If the host has already experienced a PSOD, NSX preparation is expected to complete after the reboot.