Network adapter (vmnic) is down or fails with a Failed Criteria Code

Products

VMware vSphere ESXi

Issue/Introduction

This article provides information on troubleshooting issues when a network adapter fails.

The ESXi/vCenter UI and ESXi logs show NIC adapter alerts and messages.

This KB goes over typical checks that can be done for troubleshooting.

Impact/Risks:

Packet flow associated with the services associated with the affected portgroup (either a standard switch portgroup, or a distributed virtual switch (DVS) portgroup) will cease along the data path associated with the named physical uplink (vmnic).

This will impact one or more of the following:

Virtual machines
vmkernel network adapters (including services like Management, vMotion, VSAN, iSCSI, NFS, and NSX)

The trigger for an event such as "Up" or "Down" is typically an external event upstream from the physical NIC.

The first step would be to ask the team that manages the physical infrastructure external to the affected ESXi host, to investigate for reasons they may see for the event, in their switch logs for the physical switch and/or the switchport on the physical switch to which the vmnic is connected.

Regardless, the ESXi host logs will provide the opportunity for time line analysis of the events.

It is important to note that logs never reveal causes -- they reveal only effects. But possible root causes can be investigated, once a clear understanding is available as to what events the ESXi host experienced, and when.

For more information on how to collect logs, see Collecting diagnostic information for VMware ESXi

Environment

ESXi
vCenter

Cause

In some cases, a vmnic can fail because of device firmware and/or device driver issues.

Insight into this can be found typically in the /var/run/log/vmkernel.log and its historical rotations.
For further information on device firmware / drivers, please see FAQ: Recommendation for drivers/firmware

If there is no obvious log event that would suggest a device driver / firmware issue, then the next step is to ask the team that manages the physical infrastructure external to the affected ESXi host, to investigate for reasons they may see for the event, in their switch logs for the physical switch and/or the switchport on the physical switch to which the vmnic is connected.

If that team sees nothing in their logs, then you can get a time line analysis done by Broadcom Support.

The logs required will the the ESXi host logs, per Collecting diagnostic information for VMware ESXi
After the logs are collected, open a Support case, per Creating and managing Broadcom support cases
Attach the collected logs to the Case, per Uploading files to cases on the Broadcom Support Portal

In addition to logs as outlined above, useful information to submit with your Problem Statement when you open a case would be:

The name of the ESXi host(s) where you observed the symptoms
The most recent date / time and time zone when you did not observe the symptoms, prior to the symptoms you wish investigated.
What you have done, and when you have done it, following your observation of the symptoms.
The current state of the host (example: Maintenance Mode, or Not Responding when viewed from vCenter, etc.)

Resolution

To determine the cause of the failure or eliminate common NIC issues:

Check the current status of the vmnic from either the VMware vSphere Client. or the command line via SSH or a KVM server console.
- To check the status from the vSphere Client:
  1. Select the ESX host and click the Configuration tab.
  2. Click Networking.
  3. The vmnics currently assigned to virtual switches are displayed in the diagrams. If a vmnic displays a red X, that link is currently down.
- To check the status from the command line via SSH or a KVM server console, run this command:
  
  esxcli network nic list
  
  The output appears similar to this:
```
Name     PCI Device    Driver  Admin Status  Link Status   Speed  Duplex  MAC Address         MTU  Description
-------  ------------  ------  ------------  -----------  ------  ------  -----------------  ----  -----------
vmnic0   0000:01:00.0  ixgben  Up            Up             1000  Full    ec:f4:##:##:##:##  1500  Intel(R) Ethernet Controller X540-AT2
vmnic1   0000:01:00.1  ixgben  Up            Up             1000  Full    ec:f4:##:##:##:##  1500  Intel(R) Ethernet Controller X540-AT2
```
  Note: The Admin Status is the only portion of the output that ESXi controls. This is changed by using the these commands:

esxcli network nic down -n vmnicX
esxcli network nic up -n vmnicX

The Link Status column specifies the status of the link between the physical network adapter and the physical switch.

The status can be either Up or Down. If there are several network adapters, with some being up and some down, you should verify if they are intended to be connected, because in many environments, vmnics are installed, but not connected pending future plans.

.

2. Check that the vmnic referred to in the event message is still connected to the switch and configured properly:

Make sure that the network cable is still connected to the switch and to the host.
Check that the switch connected to the system is still functioning properly and has not been misconfigured. Refer to the switch documentation for details.
Check for activity between the physical switch and the vmnic. This might be indicated either by a network trace or activity LEDs.
Check that the NIC driver is up to date: Determining Network/Storage firmware and driver version in ESXi.

3. Search for the word "vmnic" in `/var/run/log/vobd.log` log file.

If you see "vmnic down" or "vmnic up" messages, the NIC may be flapping.

Note: Some NICs report the NIC link up state only, not down. If the NIC is reported as "up" and the host was not rebooting, this is an indication that the NIC is flapping and not reporting the down state to ESXi.

Time stamps suffixed with the letter "Z" (as in the example below), are in UTC, or Coordinated Universal Time. Consult credible Internet references to determine the equivalent date / time in your time zone.

Check for a failed criteria code with the vmnic messages. If there is a failed criteria code listed, please see step 4 below.

If there is no failed criteria code, and everything was checked in step 2 above, we suggest you open a case with the hardware vendor and have them investigate.

4. In the`/var/run/log/vobd.log` file, the vmnic failure may be classified with a Failed criteria code in this log. This code explains the reason for the vmnic failure.

Example:

2020-11-17T15:37:00.330Z: [netCorrelator] 4836107000843us: [vob.net.dvport.uplink.transition.down] Uplink: vmnic4 is down. Affected dvPort: ##/50 24 e2 d9 41 e2 48 58-## ## ## ## ## ## ## ##. 3 uplinks up. Failed criteria: 128

Time - Event - Uplink# - State - Port - vSwitch - # Active Uplinks left - Failed Criteria

Note: # Active Uplinks left is indication of a failover which identifies the number of active uplinks left in the teaming policy of the virtual switch.

The following are the failed criteria codes.

1 – Link speed reported by the driver (exact match for compliance)
2 – Link speed reported by the driver (equal or greater for compliance)
4 – Link duplex reported by the driver

8 – Link LACP state down
32 – Beacon probing
64 – Errors reported by the driver or hardware
128 – Link state reported by the driver

256 – The port is blocked
512 – The driver has registered the device

Note: Failed Criteria 128 is driver reporting link state down. This can be caused by unplugging the network cable or administratively downing the physical switchport. If this was not an intended link outage it will likely be an issue with the driver, firmware, SFP+ module, cable, and/or switchport of the physical switch. Check the driver by following the below KB, and call the host hardware vendor for further troubleshooting when Failed criteria 128's are seen in the vobd log. For more information, see Determining Network/Storage firmware and driver version in ESXi.

Additional Information

The criteria that are used to determine if a network adapter in a network adapter team has failed include:

checkBeacon – By default, this check is disabled. This check becomes active when Beacon Probing is enabled on a virtual switch.
checkDuplex – By default, this check is disabled.
- If checkDuplex is true, the configured duplex mode is fullDuplex and the link is considered to be bad if the link duplex reported by driver is not the same as fullDuplex
- If checkDuplex is false, fullDuplex is unused and the link duplexity is not used as a detection method.
checkErrorPercent – By default, this check is disabled.
- If checkErrorPercent is true, the percentage mentioned in the criteria is the configured error percentage that is tolerated. The link is considered to be bad if error rate exceeds percentage.
- If checkErrorPercent is false, percentage is unused, and error percentage is not used as a detection method.
checkSpeed – The default setting is Minimum and has a default value of 10Mbps.

To use link speed as the criteria, checkSpeed must be one of these values:
- exact – Use exact speed to detect link failure. Speed is the configured exact speed in megabits per second.
- minimum – Use minimum speed to detect failure. Speed is the configured minimum speed in megabits per second.
- empty string – Do not use link speed to detect failure. Speed is unused in this case.

The Failed criteria code of 32 indicates the link has failed due to Beacon Probing detecting a problem. Beacon Probing sends beacons per VLAN between physical NICs in a team. When these are not received by other NICs this means that there is a problem in the physical network.

Note: The failure codes are accumulative so can be added together when multiple criteria are met.

When there are multiple failures, you see entries similar to these in the /var/run/log/vobd.log file:

2012-04-05T11:22:10.449Z: [netCorrelator] 1123644995238us: [vob.net.pg.uplink.transition.down] Uplink: vmnic3 is down. Affected portgroup: ########. 0 uplinks up. Failed criteria: 130

The failed criteria here is 130, which is 2 + 128. This is a combination of these two failure codes:
Link speed reported by the driver (equal or greater for compliance)
Link state reported by the driver