Slower transfer speed when migrating to different datastore type.

Products

• VMware vSAN
• VMware vSphere ESXi

Issue/Introduction

Storage vMotion migration speeds are multiple times slower than migrating to the same storage type. 

The bandwidth ranges between ~500-700 MB/s when migrating VMDKs between different datastore types (NFS to FC/vSAN) though the physical NIC speed is higher.

Environment

vSAN

ESXi

Cause

When migrating to a different storage type and format, VAAI (VMware vStorage APIs for Array Integration) is not used. There is also factors involved with the overhead of rebuilding the data on the new format. Examples are vSAN to vSAN will be faster, than vSAN to SAN. Or going SAN to SAN than vSAN to SAN. VAAI allows offloading operations, which will in terms reflect in a faster operation.

As defined in the FAQ for VAAI. VAAI hardware offload cannot be used when:

• The source and destination VMFS volumes have different block sizes
• The source and destination datastores are of different type (NFS to vSAN/FC)
• The source file type is RDM and the destination file type is non-RDM (regular file)
• The source VMDK type is eagerzeroedthick and the destination VMDK type is thin
• The source or destination VMDK is any kind of sparse or hosted format
• Cloning a virtual machine that has snapshots because this process involves consolidating the snapshots into the virtual disks of the target virtual machine.
  The logical address and/or transfer length in the requested operation is not aligned to the minimum alignment required by the storage device (all datastores created with the vSphere Client are aligned automatically)
• The VMFS datastore has multiple LUNs/extents spread across different arrays

Notes:

• When using enhanced vMotion (move both VM and datastore simultaneously), the source and destination hosts must both see the source and destination datastores, or the file transfer defaults to NFC over the the management network, instead of VAAI.
• Hardware cloning between arrays (even if within the same VMFS datastore) does not work.

When VAAI is not used, this reverts to an older and slower method called NFC (Network File Copy). 

Performance and Network Characteristics

• Bandwidth Limit: NFC is generally limited to 1Gbps by design. 
• Connectivity: Requires bidirectional connectivity between ESXi hosts over TCP port 902.
• Protocol Design: NFC is single-threaded, encrypted (SSL), and synchronous, meaning it performs one READ/WRITE/CHECK operation at a time.

 

Resolution

This is as intended, where copy speeds are slowing when going to a different storage solution or a different datastore on a complete different array that is not shared. 

When VAAI cannot be used, this will revert to NFC (Network File Copy), where this is a slower method of transfer. NFC uses Management network by default if, provisioning TCP/IP stack or provisioning service on vmotion network is not configured. 

Additional Information

Frequently Asked Questions for vStorage APIs for Array Integration