Slow virtual machine backups on ESXi

Products

VMware vSphere ESXi

Issue/Introduction

Virtual machine backups run slowly, exceed their backup window, or fail during the read of large virtual disks. Backups of large file-server or database virtual machines are the most affected. This applies to any image-level backup that reads virtual machine disks through the ESXi data path, regardless of the backup product or the storage type behind the datastore.

Backup jobs may show low throughput, and the backup application may report the source, the network, or throttling as the bottleneck. Large sequential reads may be slow or may fail while smaller operations succeed. Slow backups usually come from one or more contributing factors in the host, the network path, the storage, or the virtual machine configuration, and these factors often occur together.

Environment

VMware vSphere ESXi
Image-level backups that read virtual machine disks through the ESXi data path
Any datastore storage type and any backup product

Cause

This type of issue can be caused by several different problems, or by only one, and in practice these problems tend to stack on top of each other.

Slow backups can come from any one of several layers, and more than one factor can be present at the same time. Because the symptom alone does not identify which layer is responsible, determining the overall health requires auditing through each point in the path to confirm it is healthy: the host, the individual virtual machines, the network path, the physical adapters and switch, the storage, and the virtual machine network configuration. Contributing factors include host or VM CPU or memory contention, important traffic types sharing one network path, physical adapters or switch ports not delivering their rated speed, storage that is slow to serve the reads, a virtual network adapter that limits throughput, or a maximum transmission unit (MTU) that is either untuned for the traffic or changed on some devices in a path but not others. Any one of these can be enough to cause the symptom. Because of this, the investigation works through each layer to identify which factors apply rather than assuming a single cause.

For example, many environments run a simple default setup where management, virtual machine, storage, and backup traffic all ride the same VMkernel port on a single VLAN and uplink. Under load, a long-running backup then competes with production and storage traffic on the same path, and throughput drops for every workload on it. As another example, where an MTU is changed on some devices in a path but not others, frames larger than the smallest MTU in the path are fragmented or dropped, which adds latency rather than reducing it.

Resolution

The following are recommendations to consider in order to optimize the environment when slow backups are observed. They are not required changes. Audit each layer in the order below to find which factors apply, and follow the linked article for the procedure at each step.

Check CPU and memory health at the host and VM level

Confirm neither the host nor the individual virtual machines are resource-constrained, since CPU or memory pressure at either level slows the backup. At the host level, check for overcommitment. At the VM level, check the affected virtual machine and the backup proxy virtual machine for CPU ready or co-stop time, memory contention, and any CPU or memory limit set on the VM, since a single VM can be constrained even when the host is healthy. See Troubleshooting ESX/ESXi virtual machine performance issues.

Confirm where any storage latency is occurring

Check for latency on both sides of the path. Review the ESXi host logs for storage device latency, and review the storage's own monitoring for the same period. When the host reports high latency but the storage reports low service latency for the same window, the delay is on the path between them rather than on the storage media. See "performance has deteriorated" messages in ESXi host logs for characterizing the latency and Using esxtop to identify storage performance issues for ESXi for measuring it from the host.

Test what the network path is actually capable of

Run an iperf3 test across the relevant segment to confirm the physical adapters and switch ports deliver their rated speed, rather than assuming they do. Structure the test in two stages to isolate where any limit is:

First, test between two virtual machines on the same host and same subnet. This traffic does not leave the host, so it stays within the hypervisor and shows the true VM-to-VM ceiling with no physical network involved. Then test from a virtual machine to one on a different subnet, which forces the traffic up through the core switching and routing infrastructure. Comparing the two results shows whether throughput drops once the physical path is involved, which isolates loss in the switching or routing layer from the VM-to-VM capability. Account for the path: endpoints on the same subnet stay at layer 2, while endpoints on different subnets route through the gateway, usually a layer 3 switch or top-of-rack router, and those hops must be taken into account since each one can limit the path.

Consider separating important traffic types

Where all traffic shares a single VMkernel port, VLAN, and uplink, consider separating backup, storage, and production traffic so a long-running backup does not contend with the rest. Reducing hops also helps; placing the backup proxy on the same host that reads the source disks keeps the read path local. Separation also allows the MTU to be tuned per traffic type, which a shared port does not, since storage and vMotion benefit from a larger MTU while management does not.

Check adapter speeds, physical and virtual

Confirm the physical adapters are negotiating their expected link speed with no duplex or speed mismatch on the switch ports. Also check the virtual adapter type on the affected VMs, since it sets the throughput ceiling in the guest; VMXNET3 is the appropriate choice for high-throughput workloads, while the emulated E1000 and E1000E adapters are mainly for guests that lack a VMXNET3 driver. See Choosing a network adapter for your virtual machine.

Consider tuning MTU for the traffic type, if warranted

If the traffic type would benefit from a larger frame size, such as storage or a dedicated backup network carrying large sequential transfers, consider jumbo frames. This is optional; a default MTU is not wrong on its own, and it is common to find a consistent default that simply has not been tuned. If jumbo frames are enabled, they must be set consistently on every hop, the VMkernel port, the virtual switch, the physical switch ports, and the endpoint, and applied as a coordinated change rather than on the host alone, since a single device left at a smaller MTU fragments or drops the larger frames. See iSCSI and Jumbo Frames configuration on VMware ESXi host and Enabling Jumbo Frames on virtual switches, or for the current release Enabling jumbo frames and Using jumbo frames with iSCSI and iSER with ESXi. After any change, verify jumbo frames pass end-to-end before relying on them, using the do-not-fragment ping test in Troubleshooting disk latency when using Jumbo Frames with iSCSI or NFS datastores.

If the issue persists after working through these recommendations, contact Broadcom Support for further assistance.

Additional Information

All referenced articles: