Any HPE Server Running VMware ESXi 6.x/7.0 with certain HPE Emulex network adapters can fail with error message "F/W indicated the UE is not recoverable"

Products

• VMware vSphere ESXi

Issue/Introduction

• The host is disconnected, fail to ping and SSH

• Emulex Corporation HP FlexFabric adapters appear offline with one of these devices:
• Emulex OneConnect OCe14000 iSCSI CNA
• Emulex OneConnect OCe11100 10Gb iSCSI Initiator
• Emulex OneConnect OCe11100 iSCSI CNA
  
  
• In the /var/log/vmkernel.log file, you see similar to:

YYYY-MM-DDTHH:MM:SS.843Z cpu103:1001395041)WARNING: elxnet: elxnet_detectDumpUe:441: [vmnic1] TPE recovery: F/W indicated the UE is not recoverable (=0xd00). Taking normal UE path
YYYY-MM-DDTHH:MM:SS.242Z cpu626:1001395026)WARNING: elxnet: elxnet_detectDumpUe:441: [vmnic0] TPE recovery: F/W indicated the UE is not recoverable (=0xd00). Taking normal UE path

• Checking the serial line logging, find elxnet and elxiscsi report many warning information as follows

YYYY-MM-DDTHH:MM:SS.890Z cpu51:2102499)WARNING: elxnet: elxnet_workerThread:2113: [vmnic8] GetStats: MCC cmd timed out. timeout:36
YYYY-MM-DDTHH:MM:SS.891Z cpu51:2102499)WARNING: elxnet: elxnet_generateUE:60: [vmnic8] Injecting fatal error for post-mortem dump
YYYY-MM-DDTHH:MM:SS.017Z cpu480:2102514)WARNING: elxnet: elxnet_detectDumpUe:345: [vmnic9] UE Detected!!
ue_status_lo=0x0 ue_status_hi=0x1
YYYY-MM-DDTHH:MM:SS.017Z cpu480:2102514)WARNING: elxnet: elxnet_detectDumpUe:362: [vmnic9] UE hi: LPCMEMHOST bit set
YYYY-MM-DDTHH:MM:SS.265Z cpu580:2102528)WARNING: [8-0:85:00.2] elxiscsi_UeDetected:97 UE Detected: ue_status_lo=0x0 ue_status_hi=0x1
YYYY-MM-DDTHH:MM:SS.266Z cpu580:2102528)WARNING: [8-0:85:00.2] elxiscsi_UeDetected:98 UE Detected at 362436074 ms
YYYY-MM-DDTHH:MM:SS.266Z cpu580:2102528)WARNING: [8-0:85:00.2] elxiscsi_UeDetected:117 BG_0: UE_HIGH LPCMEMHOST bit set
YYYY-MM-DDTHH:MM:SS.266Z cpu580:2102528)WARNING: [8-0:85:00.2] elxiscsi_DetectUe:169 TPE detected: Driver and FW support TPE recovery. Moving to TPE_UE_DETECTED state

Note: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware ESXi

Cause

This is caused by a known firmware issue in HPE Emulex Network Adapters running 11.1.x,11.4.x and  12.0.x elxnet driver.

Resolution

To resolve this issue, please contact HPE.

Additional Information

For more information also refer to HPE customer advisory

Note that a reboot is required  post performing the resolution steps mentioned in the above document.