Thinly provisioned Virtual Disks inflates to a larger size during snapshot removal process

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
Disk usage of a thin-provisioned virtual disk could increase if the following sequence of steps are executed:

1. Create a SEsparse snapshot on virtual disk larger than 2TB on a VMFS5 datastore or default snapshot on VMFS6 datastore.
2. Guest Windows OS does Disk Optimization.
3. Remove the snapshot.

Due to this, following symptoms can be seen:

• Thinly provisioned Virtual Disk's size inflates to a larger size after a backup job is complete.
• Snapshot removal process takes a long time than usual.
• A TRIM operation need to be run to reclaim space from within the Guest OS.

Environment

VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5

Cause

This is the limitation of the current design of SEsparse. Unmap requests that the guest OS issues during disk optimization is processed by SEsparse. When the snapshot is removed the delta disk is consolidated to its parent which could be the base disk. During consolidation, unmaps that were received by SeSparse are not passed down to the base disk. These are handled as zero filled data. This causes to increase the disk usage of the virtual disk.

Resolution

For versions prior to vSphere 7.0 U1:

Disable automatic disk optimization.

Procedure:

1. In vSphere web client , select the affected virtual machine and select Open Console.
2. Log in as an administrator.
3. Click Start and type "Defragment and Optimize Drives" in the Search box.
4. Select "Defragment and Optimize Drives".
5. Select the drives listed in the "Optimize Drives" window that popped up.
6. Click "Change Settings" button under "Scheduled optimization".
7. Deselect "Run on a schedule (recommended)" and click OK.

 
Note: The Disk Optimization can be schedule on a Daily / Weekly or a Monthly basis, hence the frequency of the issue can depend on this settings.

If the disk usage of a thin-provisioned virtual disk increases, it can be freed by following steps:

• Execute Disk Optimization without any snapshots.
• Steps in KB 2004155 Resolution. 

For versions on vSphere 7.0 U1:

vSphere 7.0 U1 release has optimized the SESparse snapshot process to reduce bloat. "When using thin VMDKs, there can be an increase in disk usage when consolidating snapshots and unmapping the deleted data. This optimizes the consolidation process."

This optimization resolves the underlying storage and therefore, all Guest OSes shall be optimized. 

Note: Please create a ticket with VMware if the issue is seen on vSphere 7.0U1.

Additional Information

There have been some major enhancements and modifications in the functionality of defrag in Windows server 2012. 

/D     Perform traditional defrag (this is the default).
/K     Perform slab consolidation on the specified volumes.
/L     Perform retrim on the specified volumes.
/O     Perform the proper optimization for each media type.

The Storage Optimizer in Windows 8/Server 2012 , also takes care of maintenance activities like compacting data and compaction of file system allocation for enabling capacity reclamation on thinly provisioned disks. This is again platform specific, so if your storage platform supports it, Storage Optimizer will consolidate lightly used ‘slabs’ of storage and release those freed storage ‘slabs’ back to your storage pool for use by other Spaces or LUNs. This activity is done on a periodic basis i.e., without any user intervention and completes the scheduled task provided it is not interrupted by the user. 

Microsoft link: https://blogs.technet.microsoft.com/askcore/2014/02/17/whats-new-in-defrag-for-windows-server-20122012r2/

Impact/Risks:
The available free space of a Datastore fills up quickly and run out of space. This may impact already powered on VMs or not allowing VMs to be powered on