Network uplink redundancy lost due to vmnic# down

Products

• VMware vSphere ESXi

Issue/Introduction

• NIC adapter alerts/alarms in vCenter UI, such as "Network uplink redundancy degraded," "Network uplink redundancy lost," "Uplink redundancy on DVPorts degraded" or "Lost Network Redundancy."
  
  
• vmnic# showing down in the NIC list on the host:

  [root@ESXi-host:~] esxcli network nic list
  Name    PCI Device    Driver  Admin Status  Link Status  Speed  Duplex  MAC Address        MTU   Description
  ------  ------------  ------  ------------  -----------  -----  ------  -----------------  ----  -----------
  vmnic0  0000:19:00.0  i40en   Up            Up           10000  Full    ##:##:##:##:##:##  1500  Intel(R) Ethernet Controller X710 for 10GbE SFP+
  vmnic1  0000:19:00.1  i40en   Up            Down             0  Half    ##:##:##:##:##:##  1500  Intel(R) Ethernet Controller X710 for 10GbE SFP+

• In /var/log/vmkernel.log, there are entries similar to

  <timestamps> In(182) vmkernel: cpu31:2097803)netschedHClk: NetSchedHClkNotify:4749: vmnic#: link down notification
  <timestamps> In(182) vmkernel: cpu31:2097803)icen: icen_LinkEvent:11524: 0000:6f:00.0: NIC link is down

  
  
  
• In /var/log/vobd.log, there are entries similar to:

  <timestamps>: [netCorrelator] 2996811237745us: [vob.net.pg.uplink.transition.down] Uplink: vmnic# is down. Affected portgroup: <portgroup-name>. 1 uplinks up. Failed criteria: 128
  <timestamps> In(14) vobd[2097572]:  [netCorrelator] 40286920426684us: [vob.net.dvport.uplink.transition.down] Uplink: vmnic# is down. Affected dvPort: ##.##. 1 uplinks up. Failed criteria: 128

Environment

VMware vSphere ESXi

Cause

• Failed Criteria 128 is the driver reporting a link state down. This can be caused by unplugging the network cable or administratively shutting down the physical switchport.
• In some scenarios, it has been observed that a physical switchport has been disabled by loop detection mechanisms, such as Cisco Ethernet Data Plane Loopback (EDPL).
• If this was not an intended link outage it will likely be an issue with the driver, firmware, SFP+ module, cable, and/or switchport of the physical switch.

Resolution

• Check the driver by referring to the following KB:
  Determining Network/Storage firmware and driver version in ESXi
• Try replacing the cable to the physical switchport.
• Contact the host and physical switch hardware vendors for further troubleshooting and investigation.

Additional Information

Network adapter (vmnic) is down or fails with a Failed Criteria Code