Any HPE Server Running VMware ESXi 6.x/7.0 with certain HPE Emulex network adapters can fail with "F/W indicated the UE is not recoverable"

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Emulex Corporation HP FlexFabric adapters appear offline with one of these devices:
• Emulex OneConnect OCe14000 iSCSI CNA
• Emulex OneConnect OCe11100 10Gb iSCSI Initiator
• Emulex OneConnect OCe11100 iSCSI CNA
• In the /var/log/vmkernel.log file, you see similar to:

2018-03-05T06:03:01.843Z cpu103:1001395041)WARNING: elxnet: elxnet_detectDumpUe:441: [vmnic1] TPE recovery: F/W indicated the UE is not recoverable (=0xd00). Taking normal UE path
2018-03-051T06:03:02.242Z cpu626:1001395026)WARNING: elxnet: elxnet_detectDumpUe:441: [vmnic0] TPE recovery: F/W indicated the UE is not recoverable (=0xd00). Taking normal UE path

 

• The host is disconnected, fail to ping and SSH
• Checking the serial line logging, find elxnet and elxiscsi report many warning information as follows

2018-03-05T06:34:26.890Z cpu51:2102499)WARNING: elxnet: elxnet_workerThread:2113: [vmnic8] GetStats: MCC cmd timed out. timeout:36
2018-03-05T06:34:26.891Z cpu51:2102499)WARNING: elxnet: elxnet_generateUE:60: [vmnic8] Injecting fatal error for post-mortem dump
2018-03-05T06:34:27.017Z cpu480:2102514)WARNING: elxnet: elxnet_detectDumpUe:345: [vmnic9] UE Detected!!
ue_status_lo=0x0 ue_status_hi=0x1
2018-03-05T06:34:27.017Z cpu480:2102514)WARNING: elxnet: elxnet_detectDumpUe:362: [vmnic9] UE hi: LPCMEMHOST bit set
2018-03-05T06:34:27.265Z cpu580:2102528)WARNING: [8-0:85:00.2] elxiscsi_UeDetected:97 UE Detected: ue_status_lo=0x0 ue_status_hi=0x1
2018-03-05T06:34:27.266Z cpu580:2102528)WARNING: [8-0:85:00.2] elxiscsi_UeDetected:98 UE Detected at 362436074 ms
2018-03-05T06:34:27.266Z cpu580:2102528)WARNING: [8-0:85:00.2] elxiscsi_UeDetected:117 BG_0: UE_HIGH LPCMEMHOST bit set
2018-03-05T06:34:27.266Z cpu580:2102528)WARNING: [8-0:85:00.2] elxiscsi_DetectUe:169 TPE detected: Driver and FW support TPE recovery. Moving to TPE_UE_DETECTED state

Note:The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Cause

This is caused by a known firmware issue in HPE Emulex Network Adapters running 11.1.x firmware and 11.4.x elxnet driver.

Resolution

To resolve this issue upgrade the driver to version elxnet 12.0.1216.4 or higher from Customer Connect
 

Important: HPE recommends updating both the driver and firmware, to ensure the system is using a valid HPE network adapter recipe

Additional Information

For more information also refer to HPE customer advisory

Impact/Risks:
Requires reboot.