" Unrecoverable OCSD Auto DMA Error " errors reported by qlnativefc driver

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Disk I/O operation failures reported by VMs.
• High Number of aborts noticed in vmfs/volume/datastoreUUID/VM/vmware.log:

YYYY-MM-DDTHH:MM.SSSZ In(05) vcpu-17 - PVSCSI: scsi0:1: aborting cmd 0x3a2
YYYY-MM-DDTHH:MM.SSSZ In(05) vcpu-17 - PVSCSI: scsi0:1: aborting cmd 0x2c5
YYYY-MM-DDTHH:MM.SSSZ In(05) vcpu-17 - PVSCSI: scsi0:1: aborting cmd 0x2d2
YYYY-MM-DDTHH:MM.SSSZ In(05) vcpu-17 - PVSCSI: scsi0:1: aborting cmd 0x379
YYYY-MM-DDTHH:MM.SSSZ In(05) vcpu-17 - PVSCSI: scsi0:1: aborting cmd 0x319
YYYY-MM-DDTHH:MM.SSSZ In(05) vcpu-13 - PVSCSI: scsi3:58: aborting cmd 0xca
YYYY-MM-DDTHH:MM.SSSZ In(05) vcpu-23 - PVSCSI: scsi1:64: aborting cmd 0x232

Environment

VMware vSphere ESXi

Qlnativefc HBA drivers

Cause

• DMA can occur when a device driver tries to access the wrong memory addresses.
  
  In the /var/run/log/vmkernel.log file, similar entries are seen: 

  YYYY-MM-DDTHH:MM.SSSZ cpu33:2222734)qlnativefc: vmhba0(3a:0.0): qlnativefcEhAbort:2769:qlnativefcEhAbort: abortCommand mbx success.
YYYY-MM-DDTHH:MM.SSSZ cpu79:2222756)qlnativefc: vmhba0(3a:0.0): qlnativefcStatusEntry:2079:C0:T17:L160 - FCP command status: 0x5-0x0 (0x8) portid=4a0741 oxid=0x78d cdb=2a0024 len=4096 rspInfo=0x0 resid=0x0 fwResid=0x0 host status = 0x8 device status $
YYYY-MM-DDTHH:MM.SSSZ cpu73:2098498)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x2a (0x45ea4bbee7c8, 2222714) to dev "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" on path "vmhba0:C0:T17:L160" Failed:
YYYY-MM-DDTHH:MM.SSSZ cpu73:2098498)NMP: nmp_ThrottleLogForDevice:3875: H:0x8 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x430e9e6156c0 CmdSN 0x3a2
YYYY-MM-DDTHH:MM.SSSZ cpu73:2098498)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" state in doubt; requested fast path state update...
YYYY-MM-DDTHH:MM.SSSZ cpu73:2098498)ScsiDeviceIO: 4115: Cmd(0x45ea4bbee7c8) 0x2a, CmdSN 0x3a2 from world 2222714 to dev "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" failed H:0x8 D:0x0 P:0x0
YYYY-MM-DDTHH:MM.SSSZ cpu35:2209638)WARNING: qlnativefc: vmhba0(3a:0.0): qlnativefcHandle_8070Aen:430:scsi(0): Unable to acquire I2C semaphore
YYYY-MM-DDTHH:MM.SSSZ cpu35:2209638)WARNING: qlnativefc: vmhba0(3a:0.0): qlnativefcHandle_8070Aen:462:scsi(0): Unrecoverable OCSD Auto DMA Error -- Stopping Auto DMA! 0001 0400 1003
YYYY-MM-DDTHH:MM.SSSZ cpu35:2209638)WARNING: qlnativefc: vmhba0(3a:0.0): qlnativefcHandle_8070Aen:430:scsi(0): Unable to acquire I2C semaphore
YYYY-MM-DDTHH:MM.SSSZ cpu35:2209638)WARNING: qlnativefc: vmhba0(3a:0.0): qlnativefcHandle_8070Aen:462:scsi(0): Unrecoverable OCSD Auto DMA Error -- Stopping Auto DMA! 0001 0400 1003
YYYY-MM-DDTHH:MM.SSSZ cpu33:2222734)qlnativefc: vmhba0(3a:0.0): qlnativefcEhAbort:2798:C0:T17:L160: Abort command succeeded -- 1

• This can occur due to various reasons like outdated drivers, outdated SSD firmware or corrupted drivers.
• This is not an issue with VMware vSphere

Resolution

• Review Driver/Firmware details. Upgrade/downgrade the driver and firmware to match with latest compatible versions.
  Refer Broadcom compatibility guide: Broadcom Compatibility Guide
• Engage the hardware vendor if DMA errors persist.