A physical NIC may be brought down due to frequent link status flapping

Products

VMware vSphere ESXi

Issue/Introduction

When a physical NIC experiences frequent link status flapping (i.e. consecutive link down and link up events), ESXi may turn off the physical NIC to avoid unnecessary link failover and network instability.
For example, a physical NIC driver may reset the physical NIC when it detects "tx Hang". A reset causes a link down/up event to be generated. If the physical NIC is itself bad, this causes periodic reset and link status flapping. The vmkernel.log log file on the ESXi host will show that one or more network cards and repeated state change between "Up" and "Down".
During link status flapping, traffic may migrate between different physical NICs. If the flapping is too frequent, you experience network issues, such as bandwidth reduction, an unstable network or ESX Host lost Network Connectivity.
You might experience filesystem issues across several VMs in a Cluster. The filesystems are being remounted as read-only

Similar logs can be noticed in vobd.log

In(14) vobd[##]:  [netCorrelator] ##us: [esx.problem.net.vmnic.linkstate.flapping] Taking down physical NIC vmnic## because the link is flapping.
In(14) vobd[##]:  [netCorrelator] ##us: [esx.problem.net.vmnic.linkstate.flapping] Taking down physical NIC vmnic## because the link is flapping.

hostd.log reports below events

In(166) Hostd[##]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 132032 : Taking down physical NIC vmnic## because the link is unstable.
In(166) Hostd[##]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 132175 : Taking down physical NIC vmnic## because the link is unstable.

Checking nic status with "esxcli network nic get -n vmnic#" command shows "Link Status: Down by linkflapping".

Example Output:

[root@########:~] esxcli network nic get -n vmnic2

Advertised Auto Negotiation: true

Advertised Link Modes: Auto, 1000BaseX/Full, 10000BaseX/Full, 40000XLPPI/Full, 25000BaseCR1/Full, 50000BaseKR2/Full, 100000BaseCR4/Full

Auto Negotiation: true

Backing DPUId: N/A

Cable Type: DA

Current Message Level: -1

Driver Info:

Bus Info: 0000:17:00:0

Driver: ####

Firmware Version: ###

Version: ###

Link Detected: false

Link Status: Down by linkflapping

Name: vmnic2

PHYAddress: 0

Pause Autonegotiate: false

Pause RX: true

Pause TX: true

Supported Ports: TP, FIBRE, DA, BACKPLANE

Supports Auto Negotiation: true

Supports Pause: true

Supports Wakeon: false

Transceiver: internal

Virtual Address: ##:##

Wakeon: None

Environment

ESXi (All Versions)

Cause

This issue occurs when the ESXi Host detects frequent link status changes (flapping) on a physical Network Interface Card (NIC).

To prevent Network instability and unnecessary link failover events, the ESXi Host automatically disables the unstable NIC.

Resolution

This protective mechanism is triggered when the number of link status changes exceeds the configured threshold in the Net.LinkFlappingThreshold parameter (default is 60 events per minute).

Common causes for link flapping include:

Hardware faults within the physical NIC.
The NIC driver resetting the device after detecting errors, such as a "tx Hang".
Physical layer issues, such as faulty cabling or switch port errors.

Work with Hardware Vendor or Network Team to validate the link/switchport to isolate and fix the flapping issue.

If the cause of link flapping is a Hardware fault and the link does not come back up even after repair, run the following command: esxcli network nic up -n vmnic##

Additional Information

The Net.LinkFlappingThreshold parameter can be adjusted. Refer to The ESXi parameter "LinkFlappingThreshold" to adjust the link flap protection mechanism

The affected vmnic may be flapping, going down and back up in a short period of time. This article goes over typical checks that can be carried out for troubleshooting.

Setting the admin state up or down for a vmnic interface on ESXi.