A vmxnet3 virtual NIC might stop transmitting data after a reconfiguration

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

A vmxnet3 virtual NIC might stop transmitting data after a reconfiguration.
In a rare situation, the race was occurred between vmxnet3 vNIC reconfiguration and TX path due to heavy traffic.
It might cause the vNIC to stop transmitting data after the reconfiguration.

• There were messages about blocking traffic on DV port and missed IO completion.
    It lead to a backtrace for the TX world.
  --vmkernel.log
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu24:######## opID=########)NetPort: 3130: blocking traffic on DV port ########-####-####-####-############
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)Net: 3918: IO Completion missed: pkt: 0x45d994228840, IOData: 0xe729e72900270000, srcPortID: ########
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)Net: 3924:    ff ff ff ff ff ff 00 50
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)Net: 3924:    56 b3 00 04 08 00 45 10
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)Net: 3924:    01 48 00 00 00 00 80 11
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)Net: 3924:    39 96 00 00 00 00 ff ff
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)Net: 3924:    ff ff 00 44 00 43 01 34
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)Net: 3924:    89 a4 01 01 06 00 4c 07
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)Net: 3937: cc 27 00 00 00 00
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)Log: 1640: Generating backtrace for ########: NetWorld-Dev-########-Tx
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)Backtrace for current CPU #39, worldID=########, fp=0x45392109f000
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)0x45392109bad0:[0x42002456ab35]LogBacktraceInt@vmkernel#nover+0xdd stack: 0x45392109bb02, 0x420024ad224c, 0x3030203732206363, 0x3020303020303020, 0x420049002030
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)0x45392109baf0:[0x420024ad224b]Pkt_LogInfo@vmkernel#nover+0x12c stack: 0x420049002030, 0x43129b487620, 0x45392109bc50, 0x0, 0x45d994228840
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)0x45392109bb50:[0x420024ad2311]Pkt_ClearAndRelease@vmkernel#nover+0xb6 stack: 0x0, 0x453900000000, 0x45d99233e640, 0x45392109bca0, 0x0
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)0x45392109bb90:[0x42002466be92]Port_IOCompleteList@vmkernel#nover+0x31b stack: 0x392109bc50, 0x0, 0x4305c7017d80, 0x1164edaca32c38, 0x45392109bc50
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)0x45392109bc20:[0x420024667d6d]PktList_DoIOCompleteLocked@vmkernel#nover+0xc6 stack: 0x45392871f600, 0x8000000000000000, 0x0, 0x0, 0x33b3f208f6
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)0x45392109bc90:[0x420024669353]PktList_IOCompleteLocked@vmkernel#nover+0x16c stack: 0x0, 0x0, 0x0, 0x430500000000, 0x0
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)0x45392109bd00:[0x42002466ce5c]Port_InputResume@vmkernel#nover+0xd1 stack: 0x1, 0x42002464b5c8, 0x4305b5402ac0, 0x4305b5402ac0, 0x45392109be00
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)0x45392109bd50:[0x4200246b1ea5]Vmxnet3VMKDevTQDoTx@vmkernel#nover+0x23a stack: 0x4305b5402ac0, 0x45394079f000, 0x600000005, 0x0, 0x4305c7017d80
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)0x45392109bf10:[0x4200246bd966]Vmxnet3VMKDev_AsyncTxPerQ@vmkernel#nover+0xcf stack: 0x100000000, 0x45392109bf88, 0x0, 0x1a00000000, 0x1
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)0x45392109bf80:[0x420024728a77]NetWorldPerDevCB@vmkernel#nover+0x188 stack: 0x0, 0x0, 0x0, 0x45392109f000, 0x45392871f100
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu39:########)0x45392109bfe0:[0x420024adc88e]CpuSched_StartWorld@vmkernel#nover+0xbf stack: 0x0, 0x420024544fb0, 0x0, 0x0, 0x0

 

• At last, a hang is detected:
  --vmkernel.log
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu28:5443278)Vmxnet3: 19294: VM_NAME.eth6,##:##:##:##:##:##, portID(########): Hang detected,numHangQ: 1, enableGen: 39
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu28:5443278)Vmxnet3: 19300: portID:########, QID: 0, next2TX: 1832, next2Comp: 1833, lastNext2TX: 1839, next2Write:566, ringSize: 2048 >inFlight: 1, delay(ms): 2211087,txStopped: 0

Environment

VMware ESXi 8.0

Resolution

This issue is resolved in the VMware ESXi 8.0 Update 3h (Build 25067014)

Additional Information

VMware ESXi 8.0 Update 3h Release Notes
https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/release-notes/esxi-update-and-patch-release-notes/vsphere-esxi-80u3h-release-notes.html