ESXi hosts and VMs become isolated from the upstream network at the same time
search cancel

ESXi hosts and VMs become isolated from the upstream network at the same time

book

Article ID: 420959

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • Multiple Hosts and all of their VMs become unreachable on the network at the same time. 
  • Hosts are in Not Responding state in vSphere
  • Hosts are unable to ping other hosts in the same cluster (Layer 2 failure) 
  • vobd.log shows link state flapping and "Failed Criteria: 128" errors: 
    • [vob.net.dvport.uplink.transition.down] Uplink: vmnic0 is down. Affected dvPort: 15/50 3b 48 79 f4 2c 14 6a-3d 9a 2d 9d 42 12 bb 55. 0 uplinks up. Failed criteria: 128
  • If the physical NICs are Intel based cards, you may see log messages in vmkernel.log such as: 

2025-12-02T22:08:50.133Z In(182) vmkernel: cpu54:2097581)i40en: indrv_UplinkReset:2835: 0000:19:00.0: vmnic0 : device reset started
2025-12-02T22:08:50.133Z In(182) vmkernel: cpu54:2097581)i40en: i40en_UpdateUplinkLinkStatus:7101: 0000:19:00.0: Link is down for device
2025-12-02T22:08:50.193Z In(182) vmkernel: cpu94:2097581)i40en: i40en_PfTxqWait:2546: 0000:19:00.0: Tx queue request timeout
2025-12-02T22:08:50.193Z Wa(180) vmkwarning: cpu94:2097581)WARNING: i40en: i40en_TxDisableEnableHw:2704: 0000:19:00.0: Tx ring 1 disable timeout
2025-12-02T22:08:50.253Z In(182) vmkernel: cpu94:2097581)i40en: i40en_PfTxqWait:2546: 0000:19:00.0: Tx queue request timeout
2025-12-02T22:08:50.253Z Wa(180) vmkwarning: cpu94:2097581)WARNING: i40en: i40en_TxDisableEnableHw:2704: 0000:19:00.0: Tx ring 2 disable timeout
2025-12-02T22:08:50.313Z In(182) vmkernel: cpu94:2097581)i40en: i40en_PfTxqWait:2546: 0000:19:00.0: Tx queue request timeout
2025-12-02T22:08:50.313Z Wa(180) vmkwarning: cpu94:2097581)WARNING: i40en: i40en_TxDisableEnableHw:2704: 0000:19:00.0: Tx ring 3 disable timeout
2025-12-02T22:08:50.372Z In(182) vmkernel: cpu94:2097581)i40en: i40en_PfTxqWait:2546: 0000:19:00.0: Tx queue request timeout
2025-12-02T22:08:50.372Z Wa(180) vmkwarning: cpu94:2097581)WARNING: i40en: i40en_TxDisableEnableHw:2704: 0000:19:00.0: Tx ring 4 disable timeout
2025-12-02T22:08:50.432Z In(182) vmkernel: cpu94:2097581)i40en: i40en_PfTxqWait:2546: 0000:19:00.0: Tx queue request timeout
2025-12-02T22:08:50.432Z Wa(180) vmkwarning: cpu94:2097581)WARNING: i40en: i40en_TxDisableEnableHw:2704: 0000:19:00.0: Tx ring 5 disable timeout
2025-12-02T22:08:50.492Z In(182) vmkernel: cpu94:2097581)i40en: i40en_PfTxqWait:2546: 0000:19:00.0: Tx queue request timeout
2025-12-02T22:08:50.492Z Wa(180) vmkwarning: cpu94:2097581)WARNING: i40en: i40en_TxDisableEnableHw:2704: 0000:19:00.0: Tx ring 6 disable timeout
2025-12-02T22:08:50.551Z In(182) vmkernel: cpu94:2097581)i40en: i40en_PfTxqWait:2546: 0000:19:00.0: Tx queue request timeout
2025-12-02T22:08:50.551Z Wa(180) vmkwarning: cpu94:2097581)WARNING: i40en: i40en_TxDisableEnableHw:2704: 0000:19:00.0: Tx ring 7 disable timeout
2025-12-02T22:08:50.715Z In(182) vmkernel: cpu54:2097581)i40en: i40en_PfTxqWait:2546: 0000:19:00.0: Tx queue request timeout
2025-12-02T22:08:50.715Z Wa(180) vmkwarning: cpu54:2097581)WARNING: i40en: i40en_TxDisableEnableHw:2704: 0000:19:00.0: Tx ring 8 disable timeout
2025-12-02T22:08:50.726Z In(182) vmkernel: cpu54:2097581)i40en: indrv_DeviceReset:2971: 0000:19:00.0: Device Resetting
2025-12-02T22:08:50.726Z Wa(180) vmkwarning: cpu54:2097581)WARNING: i40en: i40en_SetResetFlags:16502: 0000:19:00.0: Device resetting
2025-12-02T22:08:50.726Z Wa(180) vmkwarning: cpu54:2097581)WARNING: i40en: i40en_SetResetFlags:16503: 0000:19:00.0: PF reset requested
2025-12-02T22:08:50.726Z In(182) vmkernel: cpu54:2097581)i40en: indrv_Stop:2281: 0000:19:00.0: Stopping PF
2025-12-02T22:08:50.780Z In(182) vmkernel: cpu72:2097581)i40en: i40en_ResetHW:15051: 0000:19:00.0: HW Resetting
2025-12-02T22:08:50.780Z In(182) vmkernel: cpu72:2097581)i40en: i40en_PrepForReset:14192: 0000:19:00.0: Tearing down internal switch for reset
2025-12-02T22:08:50.791Z In(182) vmkernel: cpu72:2097581)i40en: i40en_ResetAndRebuild:14483: 0000:19:00.0: Rebuilding internal switch after reset
2025-12-02T22:08:50.795Z In(182) vmkernel: cpu72:2097581)i40en: i40en_ChangeEeeCapability:9729: 0000:19:00.0: The EEE is not supported on the device
2025-12-02T22:08:50.795Z In(182) vmkernel: cpu72:2097581)i40en: i40en_ResetAndRebuild:14613: 0000:19:00.0: Attempting to rebuild switch
2025-12-02T22:08:50.809Z In(182) vmkernel: cpu72:2097581)i40en: i40en_UpdateUplinkLinkStatus:7101: 0000:19:00.0: Link is up at 10000 Mbps Full Duplex for device
2025-12-02T22:08:50.809Z In(182) vmkernel: cpu72:2097581)i40en: indrv_UplinkReset:2867: 0000:19:00.0: indrv_UplinkReset : vmnic0 device reset completed

 

Environment

  • VMware vSphere ESXi 

Resolution

The above behavior is indicative of loop behavior (ie. Broadcast Storm) happening in the upstream physical network. 

Engage your physical switch vendor and/or internal network resources to determine if there is abnormal traffic at the physical switches that is being broadcasted to the ESXi hosts. 

Powered by Wolken Software