Physical network interface (vmnic) link flaps on VMware ESXi hosts

Products

• VMware vSphere ESXi

Issue/Introduction

• A physical NIC on an ESX is flapping up/down frequently.
• In the /var/log/vmkernel.log on the ESXi host, you will see below entires:
  
  vmkernel: cpu12:2098268)bnxtnet: bnxtnet_display_link:2955: [vmnicX : 0x45210fe2c000] NIC Link is down
vmkernel: cpu12:2098268)bnxtnet: bnxtnet_display_link:2951: [vmnicX : 0x45210fe2c000] NIC Link is Up, 10000 Mbps (NRZ) full duplex, Flow control: none
  
  
• The duration of the link flap may last a few seconds where no functional impact is observed or may be longer where virtual machines using the affected vmnic loose network connectivity.
• The flap lasts a few seconds so it's possible no functional impact is observed.
• Running the following command doesn't show any drops on the NIC.
  
  esxcli network nic stats get -n vmnic<X>
  
  
• NIC private statistics show Frame Check Sequence errors on the affected NICS
• NIC private statistics can be checked using the below commands:

SSH session to ESXi host:

/usr/lib/vmware/vm-support/bin/nicinfo.sh | egrep "NIC:|fcs_err_frames"

From an ESX log bundle:

egrep "NIC:|fcs_err_frames" <esx-log-bundle>/commands/nicinfo.sh.txt
NIC:  vmnic1
      rx_fcs_err_frames: 0
      tx_fcs_err_frames: 0
NIC:  vmnic2
      rx_fcs_err_frames: 0
      tx_fcs_err_frames: 0
NIC:  vmnic3
      rx_fcs_err_frames: 0
      tx_fcs_err_frames: 0
NIC:  vmnic4
      rx_fcs_err_frames: 10000
      tx_fcs_err_frames: 0

Environment

VMware vSphere ESXi

Cause

Frame Check Sequence errors indicate a cable or hardware issue.

Resolution

Engage network and hardware teams to inspect the physical cabling, switch port, and the Network Interface Card (NIC) hardware associated with the affected vmnic.