ESXi host fails with Purple Screen of Death on Nutanix servers when virtual USB network device is enabled/disabled in quick succession

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• ESXi may crash with the following back trace:

   Code start: 0x### VMK uptime: hh:mm:ss:##.###
    0x###:[0x###]PanicvPanicInt@vmkernel#nover+0x327 stack: 0x###
    0x###:[0x###]Panic_NoSave@vmkernel#nover+0x4d stack: 0x###
    0x###:[0x###]DLM_free@vmkernel#nover+0x22d stack: 0x###
    0x###:[0x###]Heap_Free@vmkernel#nover+0xba stack: 0x###
    0x###:[0x###]LROContextUninitStats@vmkernel#nover+0x25 stack: 0x0
    0x###:[0x###]LRO_Destroy@vmkernel#nover+0x6e stack: 0x###
    0x###:[0x###]NetPollCleanupWorldHelperCB@vmkernel#nover+0x42 stack: 0x###
    0x###:[0x###]HelperQueueFunc@vmkernel#nover+0x1d2 stack: 0x###
    0x###:[0x###]CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0
    0x###:[0x###]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0

• Or:

   Code start: 0x### VMK uptime: hh:mm:ss:##.###
    0x###:[0x###]MCSLockWait@vmkernel#nover+0x10f stack: 0x0
    0x###:[0x###]MCSLockWork@vmkernel#nover+0x2a stack: 0xbad0003
    0x###:[0x###]LROContextUninitStats@vmkernel#nover+0x15 stack: 0x###
    0x###:[0x###]LRO_Destroy@vmkernel#nover+0x61 stack: 0x###
    0x###:[0x###]NetPollCleanupWorldHelperCB@vmkernel#nover+0x44 stack: 0x###
    0x###:[0x###]HelperQueueFunc@vmkernel#nover+0x19d stack: 0x###
    0x###:[0x###]CpuSched_StartWorld@vmkernel@nover+0xbf stack: 0x0
    0x###:[0x###]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0

• The following port enabled and disabled messages associated to the virtual USB network device are repeatedly logged in vmkernel.log:

    Uplink: 12282: enabled port 0xxxxxxxxx with mac xx:xx:xx:xx:xx:xx
    NetPort: 1793: disabled port 0xxxxxxxxx

• This issue has been observed only on Nutanix servers, but may not be limited to them.

Environment

VMware vSphere ESXi 7.0.x

VMware vSphere ESXi 8.0.x

Cause

Nutanix LCM process may keep enable and disable virtual USB network device device and that cause double-free problem due to a race condition.

Resolution

Broadcom VMware engineering are aware of this issue and are currently investigating.

Please feel free to click on the subscribe button to receive further notifications when this article is modified or updated.

As a workaround please avoid disabling the vusb device, e.g. by the Nutanix LCM process.