why higher workloads/ IOPS were observed for the VM and validation

Products

• VMware vSphere ESXi

Issue/Introduction

The KB article explains the workload created by the VM on Datastore and tracing

Symptoms : 

 The VM may show the excessive spikes on the 

• The VM(s) in the cluster shows high IO spike in perf charts  ( VC UI > Host and cluster view > select VM >    Monitor > Overview > disk ) 

The /var/run/log/vmkwarning logs shows  :

 YYYY-MM-DDTHH:MM.SSSZ  vmkwarning: cpu48:2098204)WARNING: ScsiDeviceIO: 1779: Device naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0 performance has deteriorated. I/O latency increased from average value of 349 microseconds to 8049 microseconds.
 YYYY-MM-DDTHH:MM.SSSZ  vmkwarning: cpu48:2098194)WARNING: ScsiDeviceIO: 1779: Device naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx1 performance has deteriorated. I/O latency increased from average value of 349 microseconds to 8303 microseconds.



Environment

VMware vSphere ESXi 6.x

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

When the number of IOPS on the Storage layer are exceeded beyond capacity by the VM running in the ESXi OS, the latency and deteriorated messages displayed

Validation steps :

localcli storage vmfs extent list 

Volume Name                      VMFS UUID                       Etent Number  Device Name                            Partition
-----------------------------------------------------------------------------------------------------------------------------------
DATASTORE_1             ########-########-####-############           0       naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx1          1
DATASTORE_2             ########-########-####-############           0       naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx2          1
DATASTORE_3             ########-########-####-############           0       naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx3          1
DATASTORE_4             ########-########-####-############           0       naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx4          1

• The NAA ID can be corelated with the Datastore name 
• The NAA ID can also be found in Datastore information . vSphere UI client > Datastore view ( Select Datastore  > Configure > Device Backing ) 

Resolution

• Engage storage vendor to investigate the high DAVG values reported in ESXTOP data.
  
  
• Engage the application team to identify any increased I/O generation and work on resolving the underlying cause.

Additional Information

Refer  : performance has deteriorated" messages in ESXi host logs