Connectivity issues on ESXi hosts with Emulex FC HBAs on HPE systems

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
On an ESXi host that has an Emulex FC HBA driver (lpfc) installed and HPE Agentless Management Service (AMS) up and running after 50 days of driver being loaded, you may experience the symptoms:

• Emulex HBAs may lose connectivity to the SAN environment which might include, but are not limited to:
  
  o    failure to process the RSCN
  o    failure to rediscover the SAN after link reset
  o    any other operation which involves ELS/CT commands will not work
   
• In the /var/log/vmkernel.log file, you see entries similar to 
  
  2018-12-16T17:09:34.805Z cpu44:68685)WARNING: lpfc: lpfc_sli_issue_iocb_wait:10765: 0:0330 IOCB wake NOT set, Data x24 x0
  Or 
  2019-06-12T02:45:35.691Z cpu0:33502)WARNING: lpfc: lpfc_sli4_send_seq_to_ulp:16032: 0:2707 Ring 1 handler: Failed to allocate iocb Rctl x22 Type x1 received
  Or
  2019-06-09T12:15:40.793Z cpu56:33666)WARNING: lpfc: lpfc_sli_issue_abort:10400: 0:(0):3164 Abort failed: No iocbqs: Data: x1 x3bc x1 x8a
  
  Note: The preceding log excerpts are only examples. The date, time, and environmental variables may vary depending on your environment.

Cause

The HPE AMS tool periodically sends management commands to the lpfc driver.

In certain situations, the driver would not receive a response to the management command, and that would result in the driver not cleaning up the command resources used to process the response properly.

This leads to the driver running out of management command resources. Any management commands sent to the HBA driver will fail to be processed due to the lack of resources.

Resolution

This issue is resolved in lpfc 12.4.270.6 async driver or later versions.

• For VMware ESXi 6.5.
• For VMware ESXi 6.7.

Notes:

• Upgrading to these versions of ESXi upgrades the lpfc driver version to 12.4.270.6 or later.
  This is a known issue affecting VMware ESXi 6.0.x. Currently, there is no resolution for this version of ESXi.