VMNIC(s) of an ESXi host are showing down when added to LACP's LAG

Products

• VMware vSphere ESXi

Issue/Introduction

 

• The VMNICs on the host are reporting a 'Down' status in both the vCenter/ESXi web console and also in the CLI when executing below command -
  # esxcfg-nics -l

  
  
• The following logs are logged in the ESXi host's vmkernel.log.
  
  yyyy-mm-ddThh-mm-ss.zzz cpu7:2097549)i40en: indrv_DeviceReset:3003: Device Resetting vmnic4
yyyy-mm-ddThh-mm-ss.zzz cpu7:2097549)i40en: indrv_Stop:2275: stopping vmnic4
yyyy-mm-ddThh-mm-ss.zzz cpu21:2097893)NetPort: 1890: disabled port 0xXXXXXXXX
yyyy-mm-ddThh-mm-ss.zzz cpu21:2097893)netschedHClk: NetSchedHClkNotify:5067: vmnic4: link down notification
yyyy-mm-ddThh-mm-ss.zzz cpu21:2097893)Team.cswitch: TeamVSLACPLAGEventCB:9117: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Received event UPLINK ADD, LAG lag1/-XXXXXXXXXX, link UNKNOWN, uplink vmnic4/0xXXXXXXXX, link UNKNOWN
yyyy-mm-ddThh-mm-ss.zzz cpu42:2098461)Net: 2238: connected Shadow of vmnic4  to null config, portID 0x400001e
yyyy-mm-ddThh-mm-ss.zzz cpu42:2098461)cswitch: VSwitchPortEthFRPUpdateInt:5829: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Unblock Port 67108894
yyyy-mm-ddThh-mm-ss.zzz cpu42:2098461)Mirror.cswitch: VSwitchMirrorPortEnable:4006: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Failed to get port 0x400001e linkup flag
yyyy-mm-ddThh-mm-ss.zzz cpu42:2098461)NetPort: 1621: enabled port 0xXXXXXXXX with mac 00:50:56:xx:xx:xx
yyyy-mm-ddThh-mm-ss.zzz cpu7:2097549)netschedHClk: NetSchedHClkPortQuiesce:7601: vmnic4: received a force quiesce for port 0xXXXXXXXX

 

• The VMNIC is part of an LACP's LAG.
  
  
• When the below command is ran, the  "port state" shows as DEF

      # esxcli network vswitch dvs vmware lacp status get

       LAG ID: XXXXXXXXXX
       Mode: Passive
   
       Nic List:
            Local Information:
                  Admin Key: 0
                  Flags: SP
                  Oper Key: 0
                  Port Number: 4
                  Port Priority: 255
                  Port State: DEF,      <--- Default State
            Nic: vmnic4
            Partner Information:
                  Age: 00:00:00
                  Device ID:
                  Flags:
                  Oper Key: 0
                  Port Number: 0
                  Port Priority: 0
                  Port State:
            State: Down
           

             Local Information:
                  Admin Key: 15
                  Flags: SP
                  Oper Key: 15
                  Port Number: 3
                  Port Priority: 255
                  Port State: AGG,SYN,COL,DIST,
            Nic: vmnic8
            Partner Information:
                  Age: 00:00:19
                  Device ID: 00:23:04:ee:be:01
                  Flags: SA
                  Oper Key: 33009
                  Port Number: 8973
                  Port Priority: 32768
                  Port State: ACT,AGG,SYN,COL,DIST,
            State: Bundled

 

• Up on reverting the switchport (to which VMNIC is connected) as standalone port from the switch side, the NIC comes back as UP. 

Environment

vSphere ESXi

Resolution

This is a Physical Switch or an ESXi host's hardware related issue. Follow the below steps to resolve it

• Make sure that the status of NIC from hardware interface like iLO/iDRAC, is UP
• Make sure that the status of the switchport to which the VMNIC is plugged in is showing UP from physical switch side.
• Make sure that driver/firmware version of the NICs are compatible with each other and are running on latest versions.
• If needed, reconfigure the LACP from switch side.
• If any of the above steps doesn't resolve the issue, please contact the physical switch/hardware Vendor.

 

Additional Information

The Port State DEF suggests several potential causes, including:

• The physical link is up, but LACP has not successfully formed.
• LACP is misconfigured on one side.
• A mismatch in the LACP mode.
• A speed mismatch.