Virtual machines go into an inaccessible state "There is no more space for virtual disk <VM name>.vmdk. You might be able to continue this session by freeing disk space on the relevant volume and clicking retry"

Products

• VMware vSAN

Issue/Introduction

• Below mentioned warning will be displayed on VM summary
  There is no more space for virtual disk <VM name>.vmdk. You might be able to continue this session by freeing disk space on the relevant volume and clicking retry. Click Cancel to terminate this session.
  
  
  
  

• The vSAN datastore shows enough total free space to provision the virtual machine.

  
  
  
• The /var/run/log/vmkernel.log file on the ESXi host reports “No space left on device” error:
  YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu54:2097677)FS3DM: 3012: status No space left on device zeroing 1 extents(1048576 each)
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu54:2097677)FS3J: 3320: Cancelling txn (0x43144b284200) callerID: 0xc1d00006 due to failurepre-committing: No space left on device
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu70:2097677)BC: 608: write to vmware.log (#### ##### ##### ###### ######) 173896 bytes failed: No space left on device

Environment

VMware vSAN 6.x
VMware vSAN 7.x
VMware vSAN 8.x

Cause

The vSAN cluster has reached critical space utilization of more than 93%. While cluster-wide space appeared available, individual capacity disks had exhausted space, leading to failure in critical VM operations like power-on, snapshot handling, or I/O.

• Validate the disk utilization in the cluster for each ESXi host. Command to check disk utilization on hosts via SSH.
   

  l() { python -c "print('-' * 115)" ; };cmmds-tool find -t HOSTNAME -f json | egrep "uuid|hostname" | sed -e 's/\"content\"://g' | awk '{print $2}' | sed -e 's/[\",\},\,]//g' | xargs -n 2 | while read hostuuid hostname; do l;echo -e " hostname: $hostname --- host UUID: $hostuuid"; l;echo -e " Disk Name\t\t| Disk UUID\t\t                | Disk Usage     | Disk Capacity | Usage Percentage";l; cmmds-tool find -f json -t DISK -o $hostuuid | egrep "uuid|content" |  awk 'NR%2{printf "%s ",$0;next;}1' | sed -e 's/[\",\},\]//g' | awk '{printf $0}' | sed -e "s/uuid: /\n uuid: /g" | grep -v "maxComponents: 0"|  awk '{print $37 " " $2 " " $5}' | while read disknaa diskuuid diskcap; do if [ $diskcap > 0 ]; then diskcapused=$(cmmds-tool find -f json -t DISK_STATUS -u $diskuuid |grep content |sed -e 's/[\",\},\]//g' | awk '{print $3}'); diskperc=$(echo "$diskcapused $diskcap" | awk '{print $1/$2*100}') ; echo -en " $disknaa\t| $diskuuid\t| $diskcapused\t | $diskcap\t | $diskperc %\n"; fi ; done ; done; l

  Example of the few disks are listed below:
   Disk Name              | Disk UUID                             | Disk Usage     | Disk Capacity         | Usage Percentage
 naa.################:2 | ########-####-####-####-############  | 1800343629332  | 1800350466048         | 99.9996 %
 naa.################:2 | ########-####-####-####-############  | 1800343629332  | 1800350466048         | 99.9996 %
 naa.################:2 | ########-####-####-####-############  | 1770765397524  | 1800350466048         | 98.3567 %

Resolution

To resolve this issue, ensure that the vSAN datastore has sufficient free space to accommodate the required objects.

Increase vSAN capacity:

1. Add additional capacity to the vSAN cluster.
2. Expand existing disk groups by adding more disks.
3. Add new ESXi hosts with disks and create disk groups on them.

Workaround (for immediate resolution):

Free up existing space in the vSAN datastore:

1. Identify and delete unused or non-critical VMs or VMDKs.
2. Migrate some VMs to another cluster or to a non-vSAN datastore.
3. Clean up unassociated objects after verifying the prerequisites outlined in the document Procedures for identifying Unassociated vSAN objects.

Additional Information

For MDs less than 255 GB in size, see: Using small magnetic disks for vSAN might result in VM failures
Failed to power on virtual machine

In the vSAN default storage policy:

• Failures To Tolerate (FTT) = 1
• Stripes per object = 1
• Object Space Reservation = 0

The reported raw free space usable is less than double the new object size plus some small metadata overhead. Attempts to create an object larger than this size, fails with above outlined error.

 

For example:

• Raw free capacity = 3000 GB.
• Required VM disk size = 250 GB.
• Required raw free space = 500 GB + overhead.

However, because Object Space Reservation is 0 (thin provisioned), it may be possible to over-commit the capacity. This will introduce a risk of running out of capacity tier space as the provisioned objects grow.

 

Ensure that using the default policy, an object that is less than or equal to 255 GB in size will have 2 components (replicas) placed on separate vSAN hosts. The components for this object are not split into smaller chunks because the Stripe Width is 1. However, if the smallest free space available on any capacity tier disks is less than object size, in this case, CLOMD will attempt one or more iterations to split replicas into smaller equal stripes that best fit on the free space of the capacity tier disks.