High storage latency and ATS miscompare errors on Nimble iSCSI volumes in vSphere

Products

• VMware vSphere ESXi

Issue/Introduction

Virtual machines and datastores residing on Nimble storage arrays may experience intermittently high read/write latency, often ranging from 500 ms to 1000 ms, while the back-end storage array reports negligible latency. This discrepancy typically occurs during automated workloads involving high-concurrency operations such as massive snapshot deletions, cloning, or large-scale power cycles. The host storage stack perceives this as a performance deterioration, leading to task management aborts and potential VM unresponsiveness. This article explains how VAAI ATS (Atomic Test and Set) lock contention creates these symptoms and provides the configuration adjustments required to mitigate metadata update conflicts on Nimble iSCSI volumes.

• High DAVG and KAVG metrics observed in esxtop.
• The vmkernel.log contains warnings: Device #### performance has deteriorated. I/O latency increased...
• SCSI sense data 0xe 0x1d 0x0 (Miscompare during verify operation) appears in logs.
• ESXi NMP device throttling is triggered: NMP: nmp_ThrottleLogForDevice:3893.

Environment

• VMware vSphere ESXi 7.x / 8.x
• Nimble Storage Array (iSCSI connectivity)
• High VM density (~60+ VMs per datastore)

 

 

Cause

The latency is caused by VAAI ATS (Atomic Test and Set) lock contention during high-concurrency metadata updates. When multiple hosts or VMs attempt to update metadata simultaneously, SCSI 0x89 (ATS) command miscompares (Sense Key 0xe ASC 0x1d ASCQ 0x0) occur. The ESXi host retries these aborted locks, which artificially inflates the host-perceived latency (DAVG/KAVG) despite the backend storage performance remaining healthy.

Resolution

To stabilize storage performance and reduce ATS contention, the following adjustments are implemented:

1. Distribution of automated virtual machine groups across a higher count of LUNs or datastores to decrease ATS lock concurrency.
2. Maintenance of virtual machine density at or below 60 active VMs per single datastore.
3. Implementation of staggered scheduling for batch operations, such as backup snapshots and disk provisioning tasks.
4. Verification and configuration of the Path Selection Policy (PSP) to NIMBLE_PSP_DIRECTED or Round Robin with an IOPS limit of 1.
5. Audit of NIC firmware and drivers to ensure alignment with the Broadcom Compatibility Guide.

Additional Information

• vSAN Health Service - Object Health
• ATS Miscompare issues on Nimble Storage