Intel NICs are flapping with ixgben driver 1.7.10 or later

Products

• VMware vSphere ESXi

Issue/Introduction

Intel NIC use the driver ixgben async driver version 1.7.10 or later.

When the last VM of a VLAN was power off,all NICs with "ixgben" driver in the same host will flap  once.

It is also triggered by removing a vmkernel NIC(vmk) if  it occupied the last active port in the VLAN.

The logs in ESXi host show similar to below

 vmkernel.log will show error similar to
 2018-11-09T07:05:20.524Z cpu146:33194)Net: 3654: disconnected client from port 0x3000008 
       
 vobd.log will show error similar to

2018-11-09T09:02:21.422Z: [netCorrelator] 75780034230us: [vob.net.pg.uplink.transition.down] Uplink: vmnic7 is down. Affected portgroup: Management Network. 1 uplinks up. Failed criteria: 128
2018-11-09T09:02:21.422Z: [netCorrelator] 75780034373us: [vob.net.pg.uplink.transition.down] Uplink: vmnic4 is down. Affected portgroup: SRV-101. 0 uplinks up. Failed criteria: 128
2018-11-09T09:02:21.422Z: [netCorrelator] 75780034376us: [vob.net.pg.uplink.transition.down] Uplink: vmnic4 is down. Affected portgroup: Management Network. 0 uplinks up. Failed criteria: 128

This is not seen with vmklinux based driver ixgben and also with Inbox driver

Note: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server Appliance 7.x or Above
VMware vCenter Server 7.x or Above

Cause

The driver "ixgben" triggers a "fake flapping"  to the ESXi and VLAN connection is  recovered after link flapping.

Resolution

A new parameter was added to disable the workaround in  ixgben driver version 1.7.20
It provided a method to disable the behavior of NIC link flapping.

In the release note of "ixgben 1.7.20" below is noted

"- Link flapping after powering off the VM or during vMotion.
   Workaround: use module parameter to disable the driver workaround for a kernel issue
   (lost VLAN connectivity once the last VM in a port group has been shut down).
   The kernel bug has been fixed in ESXi 6.5 U1 (and later releases), in which case the workaround can be safely disabled."

To download the 1.7.20 version of driver go to Broadcom Portal 

If ESXi  is running on  ESXi 6.0 P07 or  ESXi6.5U1  and later, it's recommended to install ixgben 1.7.20  or above.
 Then disable the "link flapping" workaround.

 # esxcli system module parameters set -m ixgben -a -p "VlanRemoveWorkaround=0,0"

 In above example, there were 2 network adapters using ixgben driver.

Workaround:
1. To workaround on environment which is not running on ESXi 6.0 P07 or  ESXi6.5U1 or later, install the driver version 1.7.20 and keep the parameter "VlanRemoveWorkaround" at default enable status.
2. Alternatively you can use the VMware Inbox driver or the vmklinux based ixgben driver.

Additional Information

Impact/Risks:
No Impact