Lost network connectivity with Emulex Card with error "Forcing Link Down as unrecoverable Error detected in chip / fw"

Products

• VMware
• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Lost network connectivity with Emulex Card with below:
  
  Forcing Link Down as unrecoverable Error detected in chip/fw.
   
• During boot up sequence, you may find the device was up:
  
  2017-03-23T21:39:41.266Z cpu26:32886)elxnet: elxnet_keyValueInit:1931: [vmnic1] Initialization of Key-Value with mgmt succeeded
2017-03-23T21:39:41.267Z cpu4:33451)Uplink: 7298: Device vmnic1 not yet opened
2017-03-23T21:39:41.267Z cpu4:33451)Uplink: 9307: Opening device vmnic1
...
2017-03-23T21:39:41.313Z cpu26:32886)elxnet: elxnet_linkStatusUpdate:646: vmnic1 : 0000:04:00.1 Link up - 10000 Mbps
   
• In VMkernel.log, you may find similar entries:
  
  2017-03-25T03:25:26.273Z cpu21:33430)WARNING: elxnet: elxnet_detectDumpUe:331: [vmnic2] UE Detected!!
2017-03-25T03:25:26.273Z cpu21:33430)elxnet: elxnet_detectDumpUe:347: 0000:04:00.0: Forcing Link Down as Unrecoverable Error detected in chip/fw.
2017-03-25T03:25:26.273Z cpu21:33430)WARNING: elxnet: elxnet_detectDumpUe:352: [vmnic2] UE lo: MPU bit set
2017-03-25T03:25:26.339Z cpu38:33444)WARNING: elxnet: elxnet_detectDumpUe:331: [vmnic1] UE Detected!!
2017-03-25T03:25:26.339Z cpu38:33444)elxnet: elxnet_detectDumpUe:347: 0000:04:00.1: Forcing Link Down as Unrecoverable Error detected in chip/fw.
2017-03-25T03:25:26.339Z cpu38:33444)WARNING: elxnet: elxnet_detectDumpUe:352: [vmnic1] UE lo: MPU bit set
2017-03-25T03:25:30.638Z cpu1:33410)WARNING: lpfc: lpfc_sli4_eratt_read:11015: 2:1423 HBA Unrecoverable error: uerr_lo_reg=0x4000020, uerr_hi_reg=0x0, ue_mask_lo_reg=0x4000000, ue_mask_hi_reg=0x0
  
  Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Converter Standalone 6.x
VMware vCenter Server 6.0.x
VMware vSphere ESXi 6.0
VMware vCenter Server Appliance 6.0.x
VMware Update Manager 6.0

Resolution

To resolve the issue ensure that the drivers and firmware are compatible. Engage hardware vendor to identify the recommended drivers/firmware.