Sudden spike in the vSAN cluster usage after initiating a storage policy change from Raid 1 to Raid 5 FTT 1.

Products

• VMware vSAN

Issue/Introduction

Symptoms:

Upon changing the storage policy for multiple VMs from Raid 1 to Raid 5, the used capacity of the vSAN datastore increases drastically.

Environment

VMware vSAN 7.x

VMware vSAN 8.x

Cause

When a policy change is made from a RAID1 Mirroring policy to a Raid 5/6 policy, the objects will have to be relayed out for this new policy change and vice versa. Transient capacity is generated when vSAN reconfigures objects for a policy change. But this can lead to a full cluster if to many objects are slated to be changed at once, especially if we're dealing with large objects. 

For example you have the following policy's

RAID-6

this will take up 150% of the VMs allocated space

FTT2 

this will take up 300%of the VMs allocated space

So If you are making a change on a VM that is 100 GBs on-disk. This VM will take up 300GBs using the FTT2 policy and once this VM has been moved to the Raid 6 policy it will consume 150GBs on disk. But during the change from the FTT2 policy to the Raid 6 policy this VM can use up to 450GBs on disk during the relay out of the components.

Resolution

VMware recommends making storage policy changes in small batches to avoid potentially filling up the vSAN datastore during the relay out of the object components. 

 

If in case, the used capacity of the vSAN cluster reaches 96-98% used, then the resync can get stuck as it does not have enough space to create objects using the new storage policy.

 

At that time, we will need to:

• Migrate the large size VMs to another datastore to release vSAN datastore off of some space.
• Add a host to this vSAN cluster to give the vSAN datastore some additional space to complete the resync operation.
• Another option if a RAID1 policy is in use is to change the policy to an FTT0/RAID0 policy removing the mirror copy to free up additional space.

Note: This is a last resort temporary option and a use at your own risk as you will no longer have any redundancy, and any failure could potentially result in data loss if the only active component resides on the disk/host failure.  

 

Kindly refer t\he is following article for more information on this issue, How vSAN handles Policy Changes between RAID1 Mirroring and RAID 5/6 Storage Policies.