vSAN svMotion caveats/considerations

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• When running a svMotion for a virtual machine from vSAN to an external storage or vice versa, you see the error "Insufficient disk space on datastore." when it appears the destination datastore has sufficient space.  As well, svMotion compatibility precheck fails with the error "the vmdk is larger than the maximum size supported by the datastore."
  
  
  
• While performing storage vmotion, selecting 'Configure Per disk' option in vSphere Client, we see the following error:
  
  Insufficient disk space on datastore '<Datastore_name>'.
Required space: ###########. Available space: ############.
  
  (Where 'Required space' would be more than 'Available space')
• While performing svmotion from one vsan datastore to onter vsan datastore below warning observed 
  
  "File [###] C###-I###-###_2.vmdk is larger than the maximum size supported by datastore 'sc-###-###-####1' s##-####.r####l.##
vSAN storage consumption would be 112.42 TB disk space and 0 B reserved Flash space"

Environment

Vmware vSAN 7.x

Vmware vSAN 8.x

Cause

• This is due to svMotion calculating the destination space required for the migration based on the storage policy in use for the VM/vmdk. This also impacts svMotion checking a vmdk is not larger than the maximum allowed 62TB size for a vSAN datastore.
• For example, if migrating a 1TB VM with an FTT=1 (Mirroring) it's consuming 2TB on vSAN so when migrating the VM to external storage it's expecting 2TB of free space even though it will only use 1TB on external storage.
• The same applies to migrating a 1 TB VM from external storage to vSAN. If using a storage policy of FTT=1 then 2TB of free space is required on vSAN to migrate the VM.
• If a vmdk is provisioned for 50TB with a RAID1 policy then svMotion will calculate it as 100TB so this will result in the Compatibility precheck failing with the error "the vmdk is larger than the maximum size supported by the datastore"
• If using RAID5 this will be 50TB x 1.33 which equals 66.5TB which is larger than the 62TB maximum allowed vmdk size. See vSAN Configuration Limits for more details.

For more information regarding vSAN FTT see the below docs:
vSAN Policies
Planning Capacity in vSAN
Demystifying Capacity Reporting in vSAN
RAID-1, RAID-5 or RAID-6 - What to choose? (see attachment)

Managing Fault Domains in vSAN Clusters
Adaptive RAID-5 Erasure Coding with the Express Storage Architecture in vSAN 8

Resolution

To work around this issue ensure there are current and valid backups of the VM/vmdk to be svMotioned and then do one of the following:

• Ensure the datastore the VM is being migrated to has the required space to accommodate the provisioned space plus the FTT in use of the storage policy
• If you have a current and valid backup of the VM/vmdk you can restore from backup to the destination datastore
• If migrating a large vmdk of 32TB or more with FTT=1 policy to another vSAN datastore change the policy to FTT=2/3 if it will allow for the size to be under the max vmdk size of 62TB otherwise you'll need to change it to FTT=0
• If going from vSAN to external storage and the datastore has enough space to accommodate the VM/vmdk provisioned space but not provisioned plus FTT follow the steps below:

1. Ensure there is plenty of free space on the vSAN datastore and if there is change the vSAN Storage Policy to FTT=0.
2. Wait for the resync to complete if going from FTT=2/3 to FTT=0
3. svMotion the virtual machine.

Note: Keep in mind following this action plan has its risks as the VM/vmdk will no longer have redundancy which is why it's important to confirm you have backups prior to performing the svMotion.

A proactive approach is to take these considerations into perspective when planning out VM deployments on vSAN.

For more information, see About the vSAN Default Storage Policy

Additional Information

Impact/Risks:
svMotion tasks may fail to complete.