Troubleshooting poor Storage Controller performance for VMware ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Virtual machines will experience high latency.
• When cloning or migrating virtual machines, the process takes much longer to complete compared to other hosts.
• Performing writes into a dump file may take a lot longer on this host.
• Storage devices connected with the slow performing Storage controller will report I/O latency events on /var/log/vmkernel.log of ESXi hosts.

WARNING: ScsiDeviceIO: 1513: Device naa.600c###########000000 performance has deteriorated. I/O latency increased from average value of 18932 microseconds to 1264835 microseconds.
WARNING: ScsiDeviceIO: 1513: Device naa.600c###########000000 performance has deteriorated. I/O latency increased from average value of 18934 microseconds to 2558039 microseconds.

Environment

• VMware ESXi 7.x
• VMware ESXi 8.x

Resolution

Use the esxtop utility to help determine where the bottleneck exists:

1. Log into the ESX/i server console.
2. Execute the command:

   esxtop

3. Press 'd' to go to the Storage controller statistics screen.
4. Look for the DAVG/cmd field (device latency). This gives you an idea of how long the ESX/i host is waiting (in milliseconds) for SCSI commands submitted to the storage to come back with a response.
   
   ADAPTR PATH                 NPTH   CMDS/s  READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd
   vmhba0 -               3   404.92    53.59   351.32     2.34     7.21    53.87     0.02    53.89     0.00
vmhba1 -               1     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00
   
   
   

• Your RAID type may have some bearing on performance, particularly if it is not extremely poor, yet lower than your expected baseline.
  The number of spindles/disks used per volume, along with the speed of the disks has an impact on performance.
• If a volume is degraded, such as having a failed drive in a RAID-5 disk set, read and write performance is severely hampered. You can check your RAID status in your array manager, or by observing the server's indicator/warning lights, when and where applicable.

For further troubleshooting, please contact your Server Hardware vendor.