Slow Storage vMotion Performance for Powered-Off VMs (Cold Migration)

Products

• VMware vSphere ESXi

Issue/Introduction

• Storage vMotion of a powered-off VM takes significantly longer than expected.
• Example: A 5.47 TB VM takes approximately 11 hours to migrate.
• Data transfer speeds are limited (e.g., ~1.03 Gbps) despite higher physical bandwidth availability (e.g., 10 GBe or 25 GBe).
• No underlying network errors, bandwidth constraints, or TCP retransmissions are detected in the environment.

Environment

VMware ESXi

Cause

This behavior occurs due to how vSphere handles Cold Migrations (migrations of powered-off VMs):

1. Traffic Path Selection: When a VM is powered-off, vSphere utilizes the Network File Copy (NFC) engine. By default, NFC traffic is routed through the Management Network (vmk0) rather than dedicated vMotion or vSAN VMkernel interfaces.
2. MTU Mismatch & Frame Overhead: In many environments, dedicated vMotion and vSAN networks are configured with Jumbo Frames (MTU 9000), while the Management Network (vmk0) remains at Standard Frames (MTU 1500).
3. Performance Impact: Because the transfer is forced onto vmk0, it is restricted to an MTU of 1500. Processing large data volumes (terabytes) with smaller packet sizes increases CPU serialization costs and packet overhead, leading to extended transfer times.

Resolution

Recommendation 1: Configure Dedicated NFC Traffic

You can isolate NFC traffic to a specific VMkernel adapter (such as the one used for vMotion) to leverage dedicated bandwidth and Jumbo Frames.

1. Navigate to the ESXi Host > Configure > Networking > VMkernel adapters.
2. Select the VMkernel adapter you wish to use (e.g., vMotion vmk).
3. Click Edit and ensure the Provisioning or vMotion service is enabled, or specifically look for NFC traffic enablement in supported vSphere versions.

Recommendation 2: Align MTU Settings

If the Management Network must be used for Cold Migrations, consider increasing the MTU to 9000 on the Management VMkernel interface and the supporting physical switch infrastructure to reduce packet overhead, provided this aligns with your organizational networking standards.

Additional Information

Please refer the following documents for further details related to Cold Migration