A physical NIC may be brought down due to frequent link status flapping

Products

• VMware vSphere ESXi

Issue/Introduction

• When a physical NIC experiences frequent link status flapping (i.e. consecutive link down and link up events), ESXi may turn off the physical NIC to avoid unnecessary link failover and network instability.
• For example, a physical NIC driver may reset the physical NIC when it detects "tx Hang". A reset causes a link down/up event to be generated. If the physical NIC is itself bad, this causes periodic reset and link status flapping. The vmkernel.log log file on the ESXi host will show that one or more network cards and repeated state change between "Up" and "Down".
• During link status flapping, traffic may migrate between different physical NICs. If the flapping is too frequent, you experience network issues, such as bandwidth reduction, an unstable network or ESX Host lost Network Connectivity.
• You might experience filesystem issues across several VMs in a Cluster. The filesystems are being remounted as read-only

 

• Similar logs can be noticed in vobd.log
  

  In(14) vobd[##]:  [netCorrelator] ##us: [esx.problem.net.vmnic.linkstate.flapping] Taking down physical NIC vmnic## because the link is flapping.
  In(14) vobd[##]:  [netCorrelator] ##us: [esx.problem.net.vmnic.linkstate.flapping] Taking down physical NIC vmnic## because the link is flapping.

• hostd.log reports below events
  

  In(166) Hostd[##]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 132032 : Taking down physical NIC vmnic## because the link is unstable.
  In(166) Hostd[##]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 132175 : Taking down physical NIC vmnic## because the link is unstable.

 

• Checking nic status with "esxcli network nic get -n vmnic#" command shows "Link Status: Down by linkflapping".

Example Output:

[root@########:~] esxcli network nic get -n vmnic2

   Advertised Auto Negotiation: true

   Advertised Link Modes: Auto, 1000BaseX/Full, 10000BaseX/Full, 40000XLPPI/Full, 25000BaseCR1/Full, 50000BaseKR2/Full, 100000BaseCR4/Full

   Auto Negotiation: true

   Backing DPUId: N/A

   Cable Type: DA

   Current Message Level: -1

   Driver Info: 

         Bus Info: 0000:17:00:0

         Driver: ####

         Firmware Version: ###

         Version: ###

   Link Detected: false

   Link Status: Down by linkflapping

   Name: vmnic2

   PHYAddress: 0

   Pause Autonegotiate: false

   Pause RX: true

   Pause TX: true

   Supported Ports: TP, FIBRE, DA, BACKPLANE

   Supports Auto Negotiation: true

   Supports Pause: true

   Supports Wakeon: false

   Transceiver: internal

   Virtual Address: ##:##

   Wakeon: None

Environment

ESXi (All Versions)

Cause

This issue occurs when the ESXi Host detects frequent link status changes (flapping) on a physical Network Interface Card (NIC).

To prevent Network instability and unnecessary link failover events, the ESXi Host automatically disables the unstable NIC.

 

Resolution

This protective mechanism is triggered when the number of link status changes exceeds the configured threshold in the Net.LinkFlappingThreshold parameter (default is 60 events per minute).

Common causes for link flapping include:

• Hardware faults within the physical NIC.
• The NIC driver resetting the device after detecting errors, such as a "tx Hang".
• Physical layer issues, such as faulty cabling or switch port errors.

 

Work with Hardware Vendor or Network Team to validate the link/switchport to isolate and fix the flapping issue.

If the cause of link flapping is a Hardware fault and the link does not come back up even after repair, run the following command: esxcli network nic up -n vmnic##


Additional Information

The  Net.LinkFlappingThreshold parameter can be adjusted. Refer to The ESXi parameter "LinkFlappingThreshold" to adjust the link flap protection mechanism