ESXi host kernel panic following vmxnet3 transmit completion timeout

Products

• VMware vSphere ESXi

Issue/Introduction

• An ESXi host running within VMware Cloud Foundation (VCF) experienced an unexpected outage and kernel crash following a total loss of network adapter responsiveness.
  

  +-----------------------------------------------------------------------------------+
  | VMware ESXi 8.0.3 [Releasebuild-<######>] x86_64                               |
  | #PF Exception 14 in world <########>:vmm9:##### IP ####### addr 0x7            |
  | PTEs:0x0;                                                                        |
  |                                                                                  |
  | Module(s) involved in panic: [vmkernel Version Releasebuild-<#####>]             |
  | cr0=<#######> cr2=0x7 cr3=<######> cr4=<#########>                               |
  | FMS=06/8f/8 uCode=<#######>                                                      |
  | frame=<######> ip=<#######> err=0x0 rflags=<######>                              |
  | rax=<############> rbx=<#######> rcx=<#######>                                   |
  | rdx=<#########> rbp=<######> rsi=<######>                                        |
  | rdi=<###> r8=0x0 r9=<#####>                                                      |
  | r10=<######> r11=<REDACTED> r12=<#####>                                          |
  | r13=<####> r14=<######> r15=0x0                                                  |
  | *PCPU202:<######>/vmn9:XXXXXX08265                                              |
  | PCPU 0: SVVVSVVUVVVVVVVSWVIVVIVVVVVVVVVUVVVVVVSVIIVVVVVSVVVS                     |
  | PCPU 64: VVVVSVVVVVVVVVVVVVVVVVVSVVVVVVVVVVVVVVSVVUVVVVVSVV                      |
  | PCPU128: VUSVVVSVVUVVVVIVVVVSSVUVVIUVVVIIVVVIVVVVVVUVVVVIVWVIV                   |
  | PCPU192: VVVVIUVSVIVSVVUVVVIVSVSVUVUSSVVVVSIVVVVIVVVVVVIVIIVVSVIVV               |
  | Code start: <######> VMK uptime: 10:13:36:38.214                                 |
  | <###>:[##########]TimerInsert@vmkernel#nover+0x61 stack: 0x0                     |
  | ########:[#########]Timer_AddTCWithLockDomain@vmkernel#nover+0x166               |
  | #########:[#########]Timer_AddHiRes@vmkernel#nover+0x58                          |
  | #########:[##########]CpuSchedVcpuHaltWork@vmkernel#nover+0x16a                  |
  | #########:[###########]VMMVMKCall_Call@vmkernel#nover+0x103                      |
  | ##########:[#########]VMKVMM_ArchEnterVMKernel@vmkernel#nover+0x21               |
  | ##########:[#######] (vnm64)                                                     |
  | ##########:[########]                                                            |
  | 0xf4240:[0x11]                                                                   |
  | base fs=0x0 gs=######### Kgs=0x0                                                 |
  | No port for remote debugger.     

   
  
• Diagnostic dump analysis confirmed that a live VMkernel dump was triggered by the vmxnet3 driver due to a transmit (TX) completion timeout.
• The live dump process locked up during execution, preventing a complete core dump from saving to storage. Prior to the failure, host logs recorded critical high-temperature warnings on physical adapters vmnic2 and vmnic3 before they became entirely unresponsive. The resulting panic screen displayed an invalid pointer (0xffffffffffffffff), indicating kernel memory and timer-wheel corruption following the driver hang

Environment

VMware ESXi / vSphere 8.X

Cause

Physical hardware thermal overload on network adapters vmnic2 and vmnic3 caused the physical network interface controller (NIC) hardware to freeze, resulting in vmxnet3 TX completion timeouts and subsequent kernel memory/timer-wheel corruption. 

Resolution

Engage the Original Equipment Manufacturer (OEM) server hardware vendor for further investigation