Error: "Unable to map SG array on path vmhba" and "lost volume access"

Products

VMware vSphere ESXi

Issue/Introduction

Introduction:

When upgrading to VMware vSphere ESXi 8.0 Update 3, administrators utilizing QLogic Fibre Channel over Ethernet (FCoE) adapters may experience severe storage (HPE Synergy) disruptions. This issue manifests as database virtual machines hanging, overall performance degradation, and applications going offline due to datastore disconnects. The ESXi host loses access to volumes, and disks appear degraded within the vSphere Client. These connectivity issues are tied directly to scatter-gather (SG) element alignment failures within the qedf driver. This issue has not been observed on hosts using Emulex adapters utilizing the lpfc driver, or QLogic adapters utilizing the qlnativefc driver

Symptoms:

Database VMs hang and overall host storage performance degrades on after upgrading version 8.0.Update 3 - Build 24022510.
Datastore disconnects cause hosted applications to fail.
Volumes and disks appear degraded.
After upgrading the ESXi to ESXi 8.0 U3, ESXi loose access to volumes or disks may appear degraded. This can impact the applications on the virtual machines.
Reviewing the /var/run/log/vmkernel.log reveals DMA mapping and IO errors:

qedf:vmhba#:qedfc_eh_abort:####:Info: IO not found Unable to map SG array on path vmhba##:C#:T#:L#. Status: DMA mapping could not be completed DMA Error: Can't meet SG element alignment.

Reviewing the /var/run/log/hostd.log reveals lost access to volumes and VMFS heartbeat timeouts:

Lost access to volume ########-########-####-########## (datastore) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.,Information,,esx.problem.vmfs.heartbeat.timedout,,Hostname,MM/DD/YYYY, HH:MM:SS"

Environment

VMware ESXi 8.0 Update 3 - Build 24022510
QLogic FastLinQ QL41xxx Series/QLogic FastLinQ QL45xxx 10/25 GbE Controller (FCoE)
qedf driver

Cause

This issue is caused by a conflict between the DMA constraints exposed by the qedf driver and the Pluggable Storage Architecture (PSA) logic. When I/O requests with large block sizes are issued, they may be split incorrectly based on these constraints, leading to DMA mapping failures and subsequent loss of volume access.

On ESXi 8.0.3 , DMA allocation optimization for large blocks controlled by the ESXi adv configure "ScsiPathSplitUseSimpleCloneBuffer" parameter.

Resolution

The issue is fixed on ESXi version 8.0 U3h - Build 25067014 & higher as per VMware ESXi 8.0 Update 3h Release Notes

See Download Broadcom products, patches and software to download this release.

Work Around

To workaround this issue, modify the SCSI path split configuration on the affected ESXi hosts.

Verify the current value of the ScsiPathSplitUseSimpleCloneBuffer setting (Default is 1):
esxcli system settings advanced list -o /Scsi/ScsiPathSplitUseSimpleCloneBuffer
   Path: /Scsi/ScsiPathSplitUseSimpleCloneBuffer
   Type: integer
   Int Value: 1 <<<<==== The default value of this config option
   Default Int Value: 1
   Min Value: 0
   Max Value: 1
   String Value:
   Default String Value:
   Valid Characters:
   Description: Use simple clone buffers to split IOs at path layer whenever possible.
   Host Specific: false
   Impact: none
Disable the setting by changing the value to 0:
esxcli system settings advanced set -o /Scsi/ScsiPathSplitUseSimpleCloneBuffer -i 0
Ensure the change is successful by running the command mentioned below:
esxcli system settings advanced list -o /Scsi/ScsiPathSplitUseSimpleCloneBuffer
   Path: /Scsi/ScsiPathSplitUseSimpleCloneBuffer
   Type: integer
   Int Value: 0 <<<<==== This value is reset to 0 after step #2
   Default Int Value: 1
   Min Value: 0
   Max Value: 1
   String Value:
   Default String Value:
   Valid Characters:
   Description: Use simple clone buffers to split IOs at path layer whenever possible.
   Host Specific: false
   Impact: none

No reboot is required and changes will be persistent through a reboot

Additional Information

This issue has not been observed on hosts using Emulex adapters (lpfc driver) or QLogic adapters utilizing the qlnativefc driver.
Subscribe to this knowledge article to get updates on this issue.
For HPE Synergy specific firmware/driver related information, consult the HPE Support Center.