Virtual Machine Performance Degradation During Inter-Array Storage vMotion Operations

Products

• VMware vSphere ESXi

Issue/Introduction

Performance degradation and increased read latency are observed during Storage vMotion operations between two distinct storage arrays. This behavior occurs because the migration cannot utilize VAAI (vSphere Storage APIs for Array Integration) primitives like XCOPY, which are typically restricted to intra-array data movement.

Symptoms:

• Increased Device Average Latency (DAVG) and Guest Average Latency (GAVG) during migration.

• Read latency spikes (e.g., reaching 3 ms or higher) that impact high I/O workloads.

• Migration throughput is noticeably slower compared to intra-array offloaded moves.

• High CPU or I/O wait times on the source ESXi host during the operation.

Environment

VMware vSphere ESXi
Cross-array (Inter-array) Storage configurations

Cause

When moving data between two separate physical arrays, the migration falls back to a Software Data Mover routine. In this state, the source ESXi host must manually read every block from the source array and write it to the destination array. This software-based copy is inherently slower than XCOPY offloading and is highly dependent on host resources, fabric health, and array performance.

Resolution

 

• Architectural Validation: Acknowledge that inter-array Storage vMotion is working as intended when it bypasses XCOPY. Performance expectations should not be matched against offloaded intra-array speeds.

• Analyze Latency Metrics: Use esxtop or vCenter Performance Charts to monitor DAVG.

  • If DAVG spikes correlate strictly with the destination array, it indicates the bottleneck is external to the ESXi host.

• Vendor Hardware Diagnostics: Engage the storage vendor to inspect the physical health of the destination array and fabric.

  • Ensure no hardware degradation (e.g., faulty SFP modules, bad ports, or CRC errors) is artificially throttling the Data Mover routine.

• Remediate Physical Layer: Replace any faulty hardware components identified by the storage vendor.

• Environment Optimization: Recognize that latency in non-offloaded moves is environment-dependent. Architectural variables such as HBA queue depths, fabric saturation, and array controller load will dictate the performance floor.