PSOD on the ESXi host during host preparation for NSX-T with "tcp_syncache.hashbase == NULL"

Products

• VMware NSX
• VMware vSphere ESXi

Issue/Introduction

ESXi host is being prepared for NSX-T.

Workloads (Virtual Machines) are running on the host during host preparation.

ESXi experiences a Purple Screen of Death (PSOD) with backtrace similar to:

tcp_syncache.hashbase == NULL
Code start: 0x4200######00 VMK uptime: ##:##:##:##.###
0x4539######30:[0x4200######27]PanicvPanicInt@vmkernel#nover+0x327 stack: 0x4539######38
0x4539######00:[0x4200######8e]Panic_vPanic@vmkernel#nover+0x23 stack: 0x41ff######80
0x4539######20:[0x4200######36]vmk_vPanic@vmkernel#nover+0xf stack: 0x102
0x4539######30:[0x4200######e3]panic@(tcpip4)#+0x4c stack: 0x4539######90 
0x4539######90:[0x4200######2b]syncache_init@(tcpip4)#+0x6b0 stack: 0x41ff######40 
0x4539######d0:[0x4200######d3]tcp_init@(tcpip4)#+0x3a8 stack: 0x7 
0x4539######20:[0x4200######be]domain_init@(tcpip4)#+0x2f stack: 0x4318######10
0x4539######40:[0x4200######12]vmk_net_instance_init@(tcpip4)#+0x463 stack: 0x2af8######78 
0x4539######f0:[0x4200######84]Tcpip_InitializeStackInstance@(tcpip4)#+0x215 stack: 0x2 
0x4539######b0:[0x4200######f7]Net_TcpipCreateStackInstance@vmkernel#nover+0xe4 stack: 0x3 
0x4539######00:[0x4200######2b]NetVsi_TcpipCreateNetinstanceSet@vmkernel#nover+0x184 stack: 0x3b 
0x4539######a0:[0x4200######dd]VSI_SetInfo@vmkernel#nover+0x2ca stack: 0x4539######a0 
0x4539######20:[0x4200######fc]UW64VMKSyscallUnpackVSI_Set@vmkernel#nover+0x21d stack: 0x4539######e0 
0x4539######e0:[0x4200######f2]User_UWVMK64SyscallHandler@vmkernel#nover+0x183 stack: 0xd783b########45 
0x4539######40:[0x4200######88]SyscallUWVMK64@vmkernel#nover+0x90 stack: 0x0

Environment

VMware vSphere ESXi

VMware NSX-T

Cause

Memory is heavily fragmented on the ESXi host and there is not enough contiguous memory space. Even if there is adequate heap space available, the lack of contiguous memory of sufficient size causes a failure to allocate heap space for the TCP/IP stack. The ESXi host cannot become an NSX Host Transport Node. The kernel panics from the memory allocation failure.

Resolution

The issue with kernel panics on memory allocation failures is fixed in ESX 9.

Workaround:

Reduce memory consumption on the ESXi host (e.g. by managing number of VMs on the host, or putting the host to Maintenance Mode).

If the host has already experienced a PSOD, NSX preparation can complete after the reboot. 

Post reboot, if the host is in "Install Failed" status in NSX with error indicating "Failed to install software on host" that includes a "DocumentException", choose to "Resolve" the error and preparation is expected to resume.