vSAN ESA Performance Degradation Due to Incompatible Non-NVMe or SAS Storage Devices

Products

• VMware vSAN

Issue/Introduction

VMware vSAN Express Storage Architecture (ESA) requires high-performance NVMe SSDs to maintain stability and low latency. Using SAS, SATA, or HDDs incorrectly marked as flash devices results in significant I/O latency, deterioration of storage pool health, and overall cluster instability. This article provides steps to identify non-compliant hardware and remediate the vSAN ESA configuration.

Symptoms: vSAN ESA environments using non-NVMe storage exhibit the following:

• Extreme I/O latency and storage pool performance deterioration.
• /var/run/log/vmkwarning.log showing high latency values, example:
  
  WARNING: ScsiDeviceIO: 1780: Device performance has deteriorated. I/O latency increased from average value of #### microseconds to #### microseconds.

• The vdq -i -H command identifies non-NVMe devices (e.g., TOSHIBA SAS SSDs) within the SingleTierDisks group.

Environment

VMware vSAN 8.x ESA

Cause

vSAN ESA has a strict NVMe-only requirement. SAS or SATA SSDs do not possess the necessary throughput or queue depth capabilities to support the ESA filesystem (VDFS) operations, leading to metadata inconsistencies and performance bottlenecks.

Resolution

1. Verify device compliance by running the following command to list devices in the storage pool:
   

   localcli vsan storagepool list

2. Check the vendor and model of the identified disks:

   localcli storage core device list -d

3. Compare the identified models against the VMware Compatibility Guide (VCG).
4. Evacuate all data from the impacted host using the Full Data Migration mode.
5. Remove the non-compliant SAS/SATA or HDD devices from the vSAN storage pool.
6. Replace the incompatible hardware with Broadcom-approved NVMe SSDs listed on the vSAN ESA VCG.
7. Re-add the new NVMe devices to the vSAN ESA storage pool.
8. Download the latest supported drivers or firmware from the Broadcom Product Download page.

Additional Information

• vSAN ESA Hardware Requirements