Identifying Storage I/O Control Throttling and Latency Events in ESXi

Products

• VMware vCenter Server
• VMware Telco Cloud Platform
• VMware Cloud Foundation

Issue/Introduction

• Verification is required to confirm if Storage I/O Control (SIOC) is being triggered. There are no built-in vCenter alarms that trigger when SIOC throttles storage devices.

Environment

vCenter: 8.0 u3

VCF: 5.x

TCP: 5.x

 

Cause

SIOC operates by design without triggering explicit vCenter alarms for its normal throttling operations. SIOC activity and actual latency deterioration are recorded directly in the ESXi host log files.

Resolution

 

• To verify if SIOC is throttling the storage devices, review the /var/log/storagerm.log file.

• Look for messages indicating queue depth adjustments.
  StorageRM[2099538]: Queue depth changed to 90 for device naa.########## latencyThreshold: 10 Cluster wide latency: 13.5
  Note: This message indicates that SIOC is throttling the queue. It does not specifically mean that the device is experiencing latency issues.

• To verify if actual storage latency issues exist, review the /var/log/vobd.log or /var/log/vmkernel.log files.

• Look for performance deterioration warnings.
  [YYYY-MM-DDTHH:MM:SS] cpu51:2098041)WARNING: ScsiDeviceIO: 513: Device naa.########## performance has deteriorated. I/O latency increased from average value of 38762 microseconds to 776315 microseconds.