Link coming online triggers vSAN network outage and VM freezing

Products

• VMware vSphere ESXi

Issue/Introduction

• VMs (Virtual Machines) freeze due to a VSAN outage
• Virtual network Failure Detection Policy is set to Link Status Only
• Failback to a vmnic from a Failover vmnic immediately occurs when the Failback vmnic physical switch uplink state is restored
• Network traffic does not flow out over the Failback vmnic through the physical switchport
• MAC addresses are not learned and network traffic is not forwarded during a short window of ~30+ seconds
• After the window of ~30+ seconds, MAC addresses are learned and network traffic begins to flow over the physical switchport

Environment

VMware vSphere ESXi

Cause

Link state can come up before the physical switchport is ready. Failback (default 100ms) may be too fast if the physical switchport is not ready to forward traffic.

Note : This is a physical switch issue. Contact the hardware vendor for additional assistance with the physical switchport.

Resolution

Set a failback TeamPolicyUpDelay (set to 100ms by default) under the esxi host(s) Configuration > Advanced Settings > Net > Net.teampolicyupdelay.

Ensure that the setting is slightly greater than the amount of time needed to properly initialize the physical switchport. 

Typical values, 40000 (40 sec) or 60000 (60 sec) or 90000 (90sec).

For example:

100ms default

Change to 40000 (40 sec)

 

 

Additional Information

TeamPolicyUpDelay defines the time, in milliseconds, that the ESX host waits after detecting that a pNIC has transitioned from a down state to an up state before the host starts using the pNIC again for network traffic. This delay helps to ensure that the pNIC has fully recovered and stabilized before it is put back into use.

Beacon probing is an other option that detects failures such as cable disconnects and physical switch power failures on the physical network. It uses this packet information along with link state to determine link failure.