Unexpected sudden increase in vSAN datastore storage utilization

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• Virtual Machines (VMs) on vSAN datastore fail to power ON with the following error: "There is no more space for virtual disk <VM name>.vmdk. You might be able to continue this session by freeing disk space on the relevant volume and clicking retry. Click Cancel to terminate this session.".

• Powered ON VMs go to a hung state.

• vSAN datastore capacity utilization had a sudden spike when validated from vSphere Client > vSAN Cluster > Monitor > vSAN - Capacity > Capacity History:

  

Environment

VMware vSAN 8.x

Cause

• Backup server (VMs) were residing on the vSAN datastore and a backup related task was triggered which can cause the VM's datastore space utilization to increase.

• Once the activity is completed, this depends on how the backup solution works, the utilized space is released.

• This is why there was a sudden spike in vSAN datastore space utilization and eventual decrease in utilization.

• The backup VM had large thin provisioned VMDKs (disks), which could grow in size as and when the data within the GuestOS was written on to these disks. The disks were over-provisioned, meaning when the thinly provisioned disks were to reach full space usage within the GuestOS, the size of the disk on vSAN would be more than the space available on the vSAN datastore.

• If VMware Aria Operations is available, the backup VM's space consumption can be validated for the timestamp of when the issue occurred. It can be seen that the VM's consumption had increased during the time of issue and then decreased when the issue resolved on it's own: (This view is not available in vSphere Client)

• This issue is not limited to the backup VMs, but can occur when there is sudden spike in space usage of any VM with large thinly provisioned disks, which had previously not consumed much space.

Resolution

To resolve this issue:

1. If the issue was caused by the backup VM, and the space utilization decreased on it's own after the backup job - Work with the backup team to change the backup schedule in such a way that it doesn't impact the vSAN datastore utilization.

2. Consider increasing size of vSAN datastore if all VMDKs across the VMs on vSAN datastore are overprovisioned - View oversubscription of storage space on vSAN datastore from vSphere Client > vSAN Cluster > Monitor > vSAN - Capacity > "What if analysis - Oversubscription".

Additional Information

Monitor vSAN Capacity