VMFS datastore heartbeat timeouts with SCSI sense data 0xb 0x4b 0xc4 on HPE 3PAR storage

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms

VMware ESXi hosts connected to HPE 3PAR storage arrays experience intermittent connectivity loss to VMFS volumes. Users may observe the following symptoms:

• vCenter Server Events:
  • Lost access to volume <DatastoreName> due to connectivity issues.
• Host Logs (/var/log/vmkernel.log):
  • Multiple instances of SCSI command failures targeting specific LUNs: ScsiDeviceIO: 4121: Cmd(0x45bb08862d08) 0x8a, CmdSN 0xffffe00139c48870 from world 2105423 to dev "naa.6000#################bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xb 0x4b 0xc4
  • Warnings from the Native Multipathing (NMP) stack indicating device state uncertainty: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60002ac..." state in doubt; requested fast path state update...
• HBA Driver Logs:
  • If using QLogic adapters, the qlnativefc driver may log frame drops: qlnativefc: vmhbaX(49:0.0): Dropped frame(s) detected (1064960 of 1048576 bytes).

Environment

• VMware ESXi 7.x,
• VMware ESXi 8.x
• Storage Array: HPE 3PAR
• Protocol: Fibre Channel (FC)

Cause

The error code 0xb 0x4b 0xc4 is a specific SCSI sense key returned by the storage target. It translates to:

• Sense Key: 0xb (Aborted Command)
• ASC/ASCQ: 0x4b/0xc4 (Data-Out Buffer Overflow)

• • On HPE 3PAR systems, this typically indicates that the storage processor received more data than expected for a given command or that the SAN transport layer is unstable.
  • This instability causes malformed Fibre Channel frames, leading the storage array to abort the I/O.
  • When these aborted commands impact VMFS heartbeat operations (which occur every 3 seconds), ESXi marks the volume offline if the I/O does not recover within a 16-second window.

Resolution

To resolve this issue, investigate the physical SAN fabric and HBA health:

1. Analyze Fabric Health: Check physical switch port statistics for CRC errors, Link Failures, or Loss of Signal. High error counts on specific ports often correlate with the frame drops observed in ESXi.

2. Verify Physical Layer Components: Inspect and, if necessary, replace SFPs and Fibre Channel cables associated with the HBA reporting the frame drops.

3. HBA Hardware Inspection: If frame drops persist on a single HBA while redundant paths remain clean, consider replacing the Host Bus Adapter (HBA).

4. Driver and Firmware Alignment: Ensure the FC HBA driver and firmware versions are listed as a supported combination on the Broadcom Compatibility Guide

5. Engage Storage Vendor: Provide the array-side logs to HPE support to confirm if "Data-Out Buffer Overflows" are being recorded on the storage processors.