ESXI Host Lockup that Required a Forced Reboot of the Host

Products

• VMware vSphere ESX 8.x

Issue/Introduction

ESXi server locked up and caused the server to become unresponsive

Observed connectivity issues with all the Linux VMs that were running on this host that required the VM to be rebooted

Environment

ESXi 8.0 Update 3

Cause

The unresponsiveness of the host and VMs was determined to be a NIC hardware issue, resulting in connectivity issues to an iSCSI datastore.

 

An error was detected on physical NIC resulting in receive issues.

vmkernel.log:2026-06-09T15:20:21Z In(182) vmkernel: T ERROR: Code=0x0E0E Reg=cag_int_status Addr=0x80300018 d1=0x0 d2=0x0

 

This NIC eventually failed.

2026-06-09T14:10:13.374Z In(14) vobd[2097669]:  [netCorrelator] 5072164960572us: [vob.net.dvport.uplink.transition.down] Uplink: vmnic3 is down. Affected dvPort: 16/50 0a b5 2c 2e 3b f6 cd-fc 4a 87 60 b0 ce ba c4. 0 uplinks up. Failed criteria: 128
2026-06-09T14:10:13.374Z In(14) vobd[2097669]:  [netCorrelator] 5072164960678us: [vob.net.vmnic.linkstate.down] vmnic vmnic3 linkstate down
2026-06-09T14:10:15.000Z In(14) vobd[2097669]:  [netCorrelator] 5072133928516us: [esx.problem.net.dvport.connectivity.lost] Lost network connectivity on DVPorts: "16/50 0a b5 2c 2e 3b f6 cd-fc 4a 87 60 b0 ce ba c4". Physical NIC vmnic3 is down.

 

This resulted in a failover of a the iSCSI datastore

2026-06-09T14:10:39.279Z In(14) vobd[2097669]:  [vmfsCorrelator] 5072158207303us: [esx.problem.vmfs.heartbeat.recovered] 66202d7c-28c7e0ff-67c1-ecf4bbee8cd4 vm-iscsi-datastore-01
2026-06-09T14:10:39.279Z In(14) vobd[2097669]:  The event ([esx.problem.vmfs.heartbeat.recovered] 665555d7c-200000f-67c1-ecfdddee8cd4 vm-iscsi-datastore-01) was sent immediately to hostd;

Resolution

Physical NIC was replaced by the hardware vendor.

Additional Information

Network adapter (vmnic) is down or fails with a Failed Criteria Code
https://knowledge.broadcom.com/external/article/318898/network-adapter-vmnic-is-down-or-fails-w.html

Virtual machines inaccessible with "State in Doubt" and "VMFS Heartbeat Timeouts" reported in logs
https://knowledge.broadcom.com/external/article/432799/vms-inaccessible-with-state-in-doubt-and.html