ESXi PSOD - triggered by deadlock when using deprecated 'vlance' vnic device on a virtual machine

Products

• VMware vSphere ESXi

Issue/Introduction

To help identify cause of this PSOD.

Symptoms:

PSOD (EXAMPLE)

@BlueScreen: NMI IPI: Panic requested by another PCPU. RIPOFF(base):RBP:CS [0xf992c(0x42002a800000):0x420055c0070a:0xf48] (Src 0x4, CPU87)
Backtrace:
PanicvPanicInt@vmkernel
Panic_WithBacktrace@vmkernel
NMI_Interrupt@vmkernel
IDTNMIWork@vmkernel
Int2_NMI@vmkernel
gate_entry@vmkernel
MCSLockSpin@vmkernel
MCSLockRWContended@vmkernel
MCS_DoAcqReadLockWithRA@vmkernel
RefCount_ReaderWait@vmkernel
Port_AcqNonexclWithVersion@vmkernel
PktList_DoIOComplete@vmkernel
PktList_IOComplete@vmkernel
Pkt_CompleteAndFree@vmkernel
VlanceVMKDev_RxDMA@vmkernel       
Net_VMMVlanceRxDMA@vmkernel    
VMMVMKCall_Call@vmkernel
VMKVMM_ArchEnterVMKernel@vmkernel

OR 

@BlueScreen: NMI IPI: Panic requested by another PCPU. RIPOFF(base):RBP:CS [0xf728f(0x420036c00000):0x420036e4a3be:0xf48] (Src 0x4, CPU78)
Backtrace (top 8 lines):
PanicvPanicInt@vmkernel
Panic_WithBacktrace@vmkernel
NMI_Interrupt@vmkernel
IDTNMIWork@vmkernel
Int2_NMI@vmkernel
gate_entry@vmkernel
RefCountBlock@vmkernel
Port_AcquireExcl@vmkernel
Saved backtrace from: pcpu 78 SpinLock spin out NMI
RefCountBlock@vmkernel
Port_AcquireExcl@vmkernel
Portset_GetPortExcl@vmkernel  
LLDPPortTimer@(lldp)
VmkTimerQueueWorldFunc@vmkernel
CpuSched_StartWorld@vmkernel
Debug_IsInitialized@vmkernel

Environment

VMware vSphere ESXi 7.0.x

Cause

The VLANCE vnic virtual device uses an older port locking mechanism.

A software deadlock can occur between port lock and vcd->lock of VLANCE on the newer vSwitch versions.

Resolution

1. Identify vms using 'vlance' vnic.
2. It is recommended to change to VMXNET3 or E1000E vnic type where possible.

Workaround

Use of the VLANCE vnic has been deprecated for some time, in favour of the VNXNET3 or E1000E adapter. It is recommended to change to VMXNET3 or E1000E vnic type where possible.

 

Additional Information

Consult the VMware HCL for information on Guest OS virtual device support.

VMware Compatibility Guide 

See also

Choosing a network adapter for your virtual machine

Impact/Risks

Host PSOD resulting in VM crash and resulting HA operations.