VM Full Clone task experiences severe delays or timeouts due to VAAI hardware offload fallback and datastore I/O contention

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Performing a full Virtual Machine clone, template deployment, or Storage vMotion task in a vSphere environment takes an unexpectedly long time to complete (often taking hours instead of minutes).

An inspection of hostd.log, vmkernel.log, and vmkwarning.log on the source/target ESXi host reveals the following progression:

1. VAAI Fallback Event: Hardware-accelerated cloning (XCOPY/Block Zero) times out on the storage array, forcing ESXi to switch to software-based Network File Copy (NFC) over the management network.
2. Per-Disk Duration Anomaly: Certain virtual disks (.vmdk) experience disproportionate copy durations (e.g., one disk accounting for >90% of the total task time).
3. Storage Latency Spikes: Severe latency spikes recorded on the target backing device/datastore, where average command response times (DAVG/cmd) spike far above standard baseline metrics (e.g., jumping from <5 ms up to 50–100+ ms).

Environment

VMware vCenter Server 7.x, 8.x

VMware Esxi 7.x, 8.x

Cause

This issue is typically caused by a two-stage failure sequence during the copy lifecycle:

1. VAAI Hardware Offload Timeout: The underlying storage array fails to process the VAAI Extended Copy (XCOPY) request within the expected timeout window. Consequently, the ESXi VMkernel cancels the hardware-accelerated operation and falls back to host-side software copy (NFC) over the network management path.
2. Concurrent I/O Contention and Queue Saturation: During the software fallback copy, concurrent high-throughput I/O operations (such as active scheduled backup jobs or large database writes) execute on the same target datastore or array controller. The overlapping write operations saturate storage controller queues, resulting in ESXi waiting on write lock commands and driving storage latency up significantly.

Resolution

To resolve this issue and ensure VM clone tasks complete within expected timeframes, apply the following remedial steps:

Step 1: Verify and Resolve Storage Array VAAI Offload Performance

Ensure hardware acceleration primitives are functioning reliably on the target storage array to prevent ESXi from falling back to software copying:

       1. Verify that VAAI primitives (Hardware Acceleration) are supported and active on the backing device:

           esxcli storage core device vaai status get -d <Device_NAA_ID>

       2. Work with your Storage Administrator to inspect array-side logs for the backing LUN/datastore to determine why VAAI XCOPY requests are timing out or being rejected.

       3. Ensure the storage array controller firmware is up to date according to the VMware Compatibility Guide.

Step 2: Eliminate Schedule Overlaps with Backup Windows

Avoid executing large-scale administrative operations (such as full VM clones, storage vMotions, or thick-disk provisioning) during active backup execution windows.

1. Review the execution schedule of active third-party backup jobs targeting the same datastores.
2. Stagger administrative clone tasks so they execute outside of production backup windows to prevent I/O queue congestion on the storage array controllers.

Additional Information

How to Monitor Storage Latency in Real-Time

During long-running clone tasks, monitor device latency directly from the ESXi shell using esxtop:

1. Establish an SSH connection to the ESXi host executing the copy task.
2. Launch esxtop and press d to switch to the Disk Device view.
3. Locate the target backing device and monitor the following latency columns:
• DAVG/cmd (Device Latency): Response time from the physical storage array. (Healthy baseline: <15-20 ms)
• KAVG/cmd (Kernel Latency): Time spent in the ESXi VMkernel queue. (Healthy baseline: <2-3 ms)
• GAVG/cmd (Guest Latency): Total latency seen by the OS (DAVG + KAVG).

Note: Persistent DAVG/cmd values above 20–30 ms indicate storage array controller saturation, path congestion, or severe underlying disk contention.