ESXi local disk performance degradation due to SAS physical link errors (Loss of DWORD Sync Count)

Products

• VMware vSphere ESXi

Issue/Introduction

An ESXi local disk or datastore is experiencing I/O latency and degraded performance as reported by third-party monitoring tools or storage performance charts.

Key observations:

1. ESXi host logs (e.g., /var/log/vmkernel.log) may not show significant software-level storage errors, indicating an underlying hardware or physical layer issue.

2. Running the command localcli storage san sas stats get returns output similar to:

   SasStat:
      Device Name: vmhba1
      Invalid DWORD Count: 0
      Running Disparity Error Count: 0
      Loss of DWORD Sync Count: 33554432
      Phy Reset Problem Count: 0
   

3. (Optional) The driver or firmware for the SAS HBA (vmhba1) may be out of date or incompatible with the running ESXi release.

Environment

VMware ESXi 8.0

Cause

The local disk controller (vmhba1) is experiencing physical layer link instability.

• Meaning of Loss of DWORD Sync Count: This counter tracks the number of times the physical layer lost 32-bit word (DWORD) synchronization with the target device/backplane, forcing the SAS controller to restart the link reset sequence.

• Counter Saturation (33,554,432): The value 33,554,432 (2^25) indicates that the hardware counter has reached its maximum saturation cap.

• Impact: While zero Invalid DWORD Count indicates that raw data corruption has not occurred, frequent physical link flapping causes the HBA to continually renegotiate speeds. This results in severe I/O latency spikes, storage path failovers, and potential VM hangs or storage timeout errors.

Resolution

To resolve this issue, engage the hardware vendor to further investigate.

Perform physical hardware troubleshooting and verify driver/firmware compatibility:

1. Drivers & Firmware Verification (Action Required)

• Verify that the installed HBA driver and firmware versions for vmhba1 are listed as compatible on the VMware Compatibility Guide (HCL).

• Engage your Hardware Vendor (OEM) to update the HBA firmware and driver to the recommended supported versions.

2. Physical Cable Inspection

• Check the SAS cable connecting vmhba1 to the disk/backplane for excessive bending, pin damage, or loose seats.

• Reseat both ends of the SAS cable, or replace it with a known-good spare.

3. Hardware & Environmental Checks

• Inspect the physical ports on the HBA card, storage backplane, and drive enclosures for dust, foreign objects, or pin damage.

• Check server thermal metrics and computer room cooling; overheating of the SAS HBA card can lead to severe signal attenuation.

4. Reset Counters and Verify

• After reseating or replacing hardware components, clear the SAS statistics counters or reboot the host.

• Monitor localcli storage san sas stats get over time. If Loss of DWORD Sync Count continues to increment rapidly, consider replacing the SAS HBA card or backplane/storage controller.

Additional Information

 

• Protocol Background: Loss of DWORD Sync Count is a physical-layer counter standard defined by the INCITS T10 SAS (Serial Attached SCSI) specification. The underlying HBA driver retrieves this value via the REPORT PHY ERROR LOG command within the SAS Serial Management Protocol (SMP).

• Saturating Counter Behavior: Per INCITS T10 standards, SAS physical error counters are designed as saturating counters (they freeze at their maximum cap, in this case 2^25, rather than wrapping around to zero) to ensure that persistent physical link faults are not obscured.

• Reference: INCITS T10 SAS Specification / SES Management Interface Guide (Search keyword: Loss of Dword Synchronization Count)