Network Connectivity Loss on ESXi 9.1.x

Products

• VMware vSphere ESXi
• VMware vSAN

Issue/Introduction

• ESXi 9.1.x hosts lose all or partial network traffic connectivity (Ping, SSH) when performing Maintenance Mode tasks, concurrent vMotions, or after a reboot depending on how the cluster network is configured.
  
  • This disconnect seems to be associated with the amount of load placed on the ESX host.
  • This issue will occur during higher load on the PNIC or when entering Maintenance Mode or after a reboot. 
• Connectivity is restored after "flapping" the physical vmnic - 

  esxcli network nic down -n vmnic#  
  esxcli network nic up -n vmnic#  

• vCenter reports the host as "Not Responding".
• The affected VMK's port-group is using the "route based on physical nic load" Load-Balancing policy. 
• Hosts in vSAN cluster become temporarily completely isolated from each other, resulting in a storage outage and VMs going into a read only state.
  
  
• This can occur after an upgrade or a new deployment of 9.1

Environment

• VMware ESXi 9.1.x

Cause

• The issue is caused by a race condition between the slow-path and Enhanced Network Stack (ENS) fast-path during Load-Based Teaming (LBT) failover.  
• When vmk moves to a new uplink, slow-path updates the physical switch and starts sending packets through the new uplink, but ENS continues sending packets over the old uplink for up to 150 seconds due to cached flow tables.  
  • This discrepancy causes "MAC flapping" and temporary network isolation due to MAC security configuration on the upstream switch.  

Resolution

This issue will be resolved in a later release of VCF 9.1.x.

Workaround:

• Set the Load-Balancing policy of all port groups to something other than "route based on physical NIC load".

Below is the procedure to change all the teaming policies in single configurations. 

1. Login to vCenter GUI.
2. Navigate to the "Networking" tab.
3. Right click the DVS, hover over "Distributed Port Group" and click "Manage Distributed Port Groups"
   
4. Select "Teaming and Failover" and click "next"
   
   
   
5. Select all port groups you wish to update the teaming policy and click "next":
   
6. Now change the LB policy to "Route based on originating virtual port" and click "next" and "finish":
   
   
   
7. Verify the port groups now show the updated LB policy:
   

Additional Information

• Subscribe to this article to be updated on fix status.
  • Subscribe to a Broadcom knowledge article by article or product