LAG transitions to DOWN state during NSX in-place upgrade.

Products

• VMware NSX

Issue/Introduction

• During an NSX in-place upgrade, Link Aggregation Group (LAG) interfaces on ESXi hosts, briefly transition to a DOWN state. This behavior can trigger connectivity alerts for Guest Virtual Machines, such as Load Balancers and Firewalls, that are sensitive to sub-second network interruptions.
• From ESXi host at the time of the in-place upgrade - /var/log/vmkernel.log shows similar logging: 

         ENS deactivation: 
    <Time> In(182) vmkernel: cpu106:70761449)ENS: 1961: Donot preserve uplink queues during in-place upgrade deactivate ENS
         ENS re-activation:
    <Time> In(182) vmkernel: cpu106:70761449)ENS: 1722: Donot preserve uplink queues during in-place upgrade re-activate ENS

• From ESXi host - /var/log/vobd.log shows the LAG going down, then up, representing a brief flap, seen for less than 1 second overall: 
  
  <Time> In(182) vmkernel: cpu19:2099333)Team.vswitch: TeamVSLACPLAGEventCB:9205: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Received event UPLINK LINK STATUS, LAG /-1507232220, link UNKNOWN, uplink vmnic6/0x84000012, link DOWN
<Time> In(182) vmkernel: cpu0:2099333)Team.vswitch: TeamVSLACPLAGEventCB:9205: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Received event UPLINK LINK STATUS, LAG /-1507232220, link UNKNOWN, uplink vmnic6/0x84000012, link UP

 

 

Environment

• VMware ESXi 8.0
• VMware NSX 4.2.x 

Cause

This behavior is by design for the Enhanced Networking Stack (ENS) and the NVIDIA `nmlx5_core` driver during an in-place upgrade. To facilitate the upgrade without a host reboot, ENS must be deactivated and re-activated to load the new modules, this triggers the NVIDIA driver to temporarily bring down the physical NICs, resulting in a brief flap of the associated LAG.

Resolution

This is a condition that may occur in a VMware NSX environment.