Connectivity issues on ESXi hosts with the FCoE module (brcmfcoe) on HPE systems

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
On an ESXi that has Emulex FCoE HBA driver (brcmfcoe) installed and HPE Agentless Management Service (AMS) up and running after 50 days of drivers being loaded, these symptoms may appear:

• Emulex HBAs may lose connectivity to the SAN environment which might include, but are not limited to:
  
  o    failure to process the RSCN
  o    failure to rediscover the SAN after link reset
  o    any other operation which involves ELS/CT commands will not work
   
• In the /var/log/vmkernel.log file, there are entries similar to:
  
  2018-12-16T17:09:34.805Z cpu44:68685)WARNING: brcmfcoe: lpfc_sli_issue_iocb_wait:10765: 0:0330 IOCB wake NOT set, Data x24 x0
  Or 
  2019-06-12T02:45:35.691Z cpu0:33502)WARNING: brcmfcoe: lpfc_sli4_send_seq_to_ulp:16032: 0:2707 Ring 1 handler: Failed to allocate iocb Rctl x22 Type x1 received
  Or
  2019-06-09T12:15:40.793Z cpu56:33666)WARNING: brcmfcoe: lpfc_sli_issue_abort:10400: 0:(0):3164 Abort failed: No iocbqs: Data: x1 x3bc x1 x8a
  
  Note: The preceding log excerpts are only examples. The date, time, and environmental variables may vary depending on the environment.

Environment

VMware vSphere ESXi 6.0

Vmware vSphere ESXI 6.5

Vmware vSphere ESXI 6.7

Cause

The HPE AMS tool periodically sends management commands to the brcmfcoe driver.

In certain situations, the driver would not receive a response to the management command, and that would result in the driver not cleaning up the command resources used to process the response properly.

This would lead to the driver running out of management command resources. Any management commands sent to the HBA driver will fail to be processed due to the lack of resources.

Resolution

This issue is resolved in brcmfcoe 12.0.1278.0 async driver or later versions.

Notes:

• Upgrading to these versions of ESXi upgrades the brcmfcoe driver version to 12.0.1278.0 or later.
  This is a known issue affecting VMware ESXi 6.0.x. Currently, there is no resolution for this version of ESXi.