TeamPolicyUpDelay timer ignored on ESXi hosts blackholing traffic
search cancel

TeamPolicyUpDelay timer ignored on ESXi hosts blackholing traffic

book

Article ID: 453207

calendar_today

Updated On:

Products

VMware NSX VMware vSphere ESX 8.x

Issue/Introduction

  • ESXi hosts enter a network isolated state and workload VMs lose network connectivity shortly after a physical switch is brought online.

  • The TeamPolicyUpDelay timer is ignored, and network connectivity is restored instantly instead of waiting for the configured delay.

  • vobd.log: Shows the physical NIC coming up and connectivity immediately restoring:

    2026-06-27T00:00:33.000Z In(14) vobd[########]: [netCorrelator] ########]us: [esx.clear.net.connectivity.restored] Network connectivity restored on virtual switch , portgroups: . Physical NIC vmnic0 is up

  • hostd.log: Shows the event manager restoring connectivity immediately:

    2026-06-27T00:00:33.001Z In(166) Hostd[########]]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 10490 : Network connectivity restored on virtual switch , portgroups: . Physical NIC vmnic0 is up.

  • vmkernel.log: Shows the driver link status change followed immediately by device UP notifications:

    2026-06-27T00:00:32.483Z In(182) vmkernel: cpu47:########])<NMLX_INF> nmlx5_core: vmnic0: nmlx5_en_UplinkLinkStateSetOS - (nmlx5_core_en_uplink.c:########]) Changing link status from DOWN Half Duplex 0 to UP Full Duplex 25000

    2026-06-27T00:00:33.351Z In(182) vmkernel: cpu42:########])NetqueueBal: 5211: vmnic0: device Up notification, reset logical space needed

Environment

  • VMware ESXi 8.0.3 
  • VMware NSX 4.2.1, 4.2.2

Cause

This is a known issue impacting VMware NSX and VMware ESXi.

A defect in ESXi’s NIC teaming implementation causes this hold-down state to be ignored when a new virtual machine network adapter connects to the vSwitch during the delay window. Each time a new port is enabled on the vSwitch — for example, when vSphere HA restarts a virtual machine, when vMotion migrates a virtual machine onto the host, or when a virtual machine is powered on — ESXi recalculates the NIC teaming policy for that port. Due to the defect, this recalculation does not check whether a given uplink is in its TeamPolicyUpDelay hold-down period and incorrectly adds the recovering uplink to the active uplink pool for the new port. Once the uplink appears in the active pool for any port, ESXi begins routing outbound traffic through it immediately, bypassing the configured hold-down delay for all ports on the vSwitch.

The failure chain can look like below:

  1. Initial State: Let's assume Management traffic is actively mapped to vmnic0.
  2. Physical Hardware Replacement: vmnic0 goes down, causing management traffic to successfully fail over to the secondary link, vmnic1.
  3. Physical Hardware Restored: Following the physical switch (SW) replacement, vmnic0 comes back online.
  4. Delay Timer Initiation: The system arms TeamPolicyUpDelay for 30 minutes to allow the newly restored link to stabilize before failing back.
  5. The Bug Trigger: While the 30-minute timer is still running, a VM port comes up (caused by manual port disable/enable, DRS, or vSphere HA). Due to an identified bug, this event bypasses TeamPolicyUpDelay.
  6. Premature Fallback: Because of the configured fallback mechanism and the bypassed timer, management traffic and a few other port groups are immediately forced back to vmnic0.
  7. Management Blackhole: The physical switch is not yet properly configured to handle the traffic, so routing it back to vmnic0 creates a management blackhole.
  8. Host Isolation: This management blackhole causes the ESXi host to become isolated from the network.
  9. Cascading Effect (vSphere HA): Host isolation triggers vSphere HA, causing VMs on the isolated host to migrate to other surviving hosts in the cluster.
  10. Cluster-Wide Outage: As migrated VMs land and their ports come up on the new hosts, the exact same bug is triggered. TeamPolicyUpDelay is bypassed everywhere, forcing management traffic to fall back to vmnic0 on all hosts in the cluster, resulting in total cluster-wide isolation.

Resolution

Fix:

The code has been patched to ensure the TeamPolicyUpDelay timer is consistently triggered and honored across all hosts when physical links transition to an UP state.

FIX DETAILS:

ESX 8.0.3 P10 | 9.1.1.0 |
NSX 9.1.1.0 | 4.2.5 | 4.2.4.1.1

Workaround:

If you are unable to upgrade and need a workaround on your current and impacted version, please open a support request with Global Support (GS) and reference this Knowledge Base article.

Powered by Wolken Software