Troubleshooting poor local storage controller performance for VMware ESXi/ESX

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• When cloning or migrating virtual machines to the local storage of an ESX/i host, the process takes much longer to complete compared to other hosts.
• Performing writes into a dump file may take a lot longer on this host.
• Performing reads from a file may take longer than other hosts, but not as long as the above write performance test.
• There may be no errors in the logs (such as /var/log/messages or /var/log/vmkernel) to follow up with.

Environment

VMware ESX Server 2.5.x
VMware ESXi 4.0.x Embedded
VMware vSphere ESXi 5.1
VMware ESX Server 3.0.x
VMware ESX 4.0.x
VMware ESXi 4.0.x Installable
VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.5
VMware ESX Server 3.5.x
VMware ESXi 4.1.x Installable
VMware ESXi 4.1.x Embedded

Resolution

To resolve this issue, use the esxtop utility to help determine where the bottleneck exists.

To determine where the bottleneck exists:

1. Log into the ESX/i server console.
2. Execute the command:
   
   esxtop
   
3. Press d to go to the disk statistics screen.
4. Press f to go to the field selector.
5. If the J does not have an asterisk (*) beside it, press j to add that field to the view. You can remove E from the view.
6. Look for the DAVG/cmd field (device latency). This gives you an idea of how long the ESX/i host is waiting (in milliseconds) for SCSI commands submitted to the storage to come back with a response.

If the commands appear to have a considerable amount of latency (more than 50 ms) for the local vmhba, device contention is being encountered.

• Write performance is degraded on RAID controllers that have a discharged or absent battery because the write cache is disabled.
  Check the battery to ensure that it is present and/or charged. Your SCSI controller may not necessarily be faulty.
  
• Your RAID type may have some bearing on performance, particularly if it is not extremely poor, yet lower than your expected baseline.
  The number of spindles/disks used per volume, along with the speed of the disks has an impact on performance.
  
• If a volume is degraded, such as having a failed drive in a RAID-5 disk set, read and write performance is severely hampered. You can check your RAID status in your array manager, or by observing the server's indicator/warning lights, when and where applicable.
  
• As a troubleshooting measure, it may be worth breaking the disks into a RAID-less configuration, to eliminate this as being a possibility. This may require a reinstallation of VMware ESX/i.

For further troubleshooting, contact your server hardware vendor.

Additional Information

For information about write back caching on Storage Area Networks (SAN), see Write-cache disabled on storage array causing performance issues or failures (1002282).

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box