Troubleshooting poor local storage controller performance for VMware ESXi/ESX

Products

• VMware vSphere ESXi

Issue/Introduction

• When cloning or migrating virtual machines to the local storage of an ESX/i host, the process takes much longer to complete compared to other hosts.
• Performing writes into a dump file may take a lot longer on this host.
• Performing reads from a file may take longer than other hosts, but not as long as the above write performance test.
• There may be no errors in the logs (such as /var/log/messages or /var/log/vmkernel) to follow up with.

Environment

• VMware ESX Server 2.5.x
• VMware ESXi 4.0.x Embedded
• VMware vSphere ESXi 5.1
• VMware ESX Server 3.0.x
• VMware ESX 4.0.x
• VMware ESXi 4.0.x Installable
• VMware vSphere ESXi 5.0
• VMware vSphere ESXi 5.5
• VMware ESX Server 3.5.x
• VMware ESXi 4.1.x Installable
• VMware ESXi 4.1.x Embedded

Resolution

Use the esxtop utility to help determine where the bottleneck exists:

1. Log into the ESX/i server console.
2. Execute the command:
   
   esxtop
3. Press d to go to the disk statistics screen.
4. Press f to go to the field selector.
5. If the J does not have an asterisk (*) beside it, press j to add that field to the view. You can remove E from the view.
6. Look for the DAVG/cmd field (device latency). This gives you an idea of how long the ESX/i host is waiting (in milliseconds) for SCSI commands submitted to the storage to come back with a response.

If the commands appear to have a considerable amount of latency (more than 50 ms) for the local vmhba, device contention is being encountered.

• Write performance is degraded on RAID controllers that have a discharged or absent battery because the write cache is disabled.
  Check the battery to ensure that it is present and/or charged. Your SCSI controller may not necessarily be faulty.
• Your RAID type may have some bearing on performance, particularly if it is not extremely poor, yet lower than your expected baseline.
  The number of spindles/disks used per volume, along with the speed of the disks has an impact on performance.
• If a volume is degraded, such as having a failed drive in a RAID-5 disk set, read and write performance is severely hampered. You can check your RAID status in your array manager, or by observing the server's indicator/warning lights, when and where applicable.
• As a troubleshooting measure, it may be worth breaking the disks into a RAID-less configuration, to eliminate this as being a possibility. This may require a reinstallation of VMware ESX/i.

For further troubleshooting, contact your server hardware vendor.