HPE HBA driver "smartpqi" reports "Out of memory"

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• In some HPE server, configured with the HPE Smart PQI controllers.
• The driver "smartpqi" was used for this HBA / IO controller.
• The ESXi host was non-responsive in the vCenter.
• The host management agent "hostd" was non-responsive.

-- vmkernel.log
2019-03-28T12:15:01.414Z cpu42:2657267)ALERT: hostd detected to be non-responsive

* The HBA driver "smartpqi" was report " "Out of memory" in the log.
-- vmkernel.log
2019-03-28T06:25:22.704Z cpu58:875065)smartpqi: pqi_AllocDmaFromMemPool:92: vmk_MemPoolAlloc failed: Out of memory.
2019-03-28T06:25:22.704Z cpu58:875065)smartpqi: os_dma_mem_alloc:195: Failed to allocate dma mem from mem pool
2019-03-28T06:25:22.704Z cpu58:875065)smartpqi: pqisrc_passthru_ioctl:56: Failed to Allocate dma mem for Ioctl PassthruCmd Buffer : -1

Environment

VMware ESXi 6.7.x
VMware ESXi 6.5.x

Cause

• The HBA driver "smartpqi" always requests the physical memory address below 4 GB range
• This fails to get additional memory If there was no any DMA memory available in that range.

As lack of memory, the 3th utility "SSA CLI" will slowly get feedback from the HBA driver.

• This will lead to host management agent "hostd" to turn non-responsive 

Resolution

• The latest driver "smartpqi" version "1.0.3", an enhancement was added to fix this issue.
• It's release on Jan 2019.
• Download VMware ESXi 6.5 smartpqi 1.0.3.2035 SAS Driver for HPE Smart PQI controllers

https://customerconnect.vmware.com/web/vmware/details?downloadGroup=DT-ESXI65-MICROSEMI-SMARTPQI-1032035&productId=614

In the release note snippet :
Fixes & Enhancements:
---------------------
* Added an enhancement to remove "Below 4 GB" DMA memory usage.
  The driver always requests the physical memory address below 4 GB range and there might not be any DMA memory available in that range.

Note: As this enhancement,it's requested to use the 64-bit SSA CLI utility to work with the latest driver.

Workaround:

• Sometimes, the host will be resume by restarting the hostd.

# /etc/init.d/hostd restart

Additional Information

Impact/Risks:

• The ESXi host will be un-responsive from the vCenter