Troubleshooting ESXi datastore or VMFS volume that is full or near capacity

Products

VMware vSphere ESXi

Issue/Introduction

Capacity Guidelines

ESXi hosts store all files necessary for virtual machine operation in a VMware Virtual Machine File System (VMFS). Proper capacity management is critical as this file system is required for any operations related to virtual machines.

This document outlines potential issues related to a VMFS file system becoming full. All VMFS file systems (VMFS2, VMFS3, VMFS5, VMFS6) must retain a certain level of free space to ensure that virtual machines function properly. Ensure that these capacity thresholds are met:

200 Megabyte Capacity	VMware suggests 200 MB as the threshold where ESXi system administrators should take action to resolve capacity issues. VMFS volumes vary widely in size and it is impossible to define a precise percentage, but the risk of unexpected problems becomes greater at this capacity.
100 Megabyte Remaining	100 MB is defined as a critical threshold at which ESXi system administrators must take immediate action to correct the issue.

For more information, see Increase VMFS Datastore Capacity, and Unable to grow or expand a VMFS volume or datastore.

Potential Issues

This is a list of some potential issues that may affect virtual machines residing on a VMFS that is full:

Virtual machine startup failure	ESXi hosts are able to provide more RAM space than is available in the total physical RAM installed in the server. This is termed overcommitment. ESXi hosts create a .vswp files for any virtual machine that has allocated memory greater than the physical RAM that is available to the virtual machine. The `<VMName>.vswp` file is created when the virtual machine is started. Without sufficient space, this file cannot be created, and the virtual machine is unable to start.
vMotion failures	ESXi hosts that are in a cluster that have VMware HA/DRS enabled may encounter failures when vMotion attempts to move a virtual machine from one host to another. It is possible for this failure to occur when it may not be immediately identified as a failure because VMware DRS can be configured to vMotion virtual machines without interaction from an administrator. Storage vMotion or migrations typically fail at 90% or higher when the destination datastore lacks sufficient space for the virtual machine or its overhead files.
Snapshot creation/commit	When a snapshot is taken or committed to the virtual machine, data is written to the VMFS file system. Snapshots require space to write new data. If the file system is full, snapshots cannot grow, does not get writtenand consolidation (committing) may fail due to lack of space for the redo logs.
Degraded performance	System commands like `ls` and other file management functions perform poorly or time out when the VMFS volume is exhausted. Though less likely to affect virtual machine availability, these issues impact ESXi system administrators and must be resolved. Note: Journaling filesystems (like VMFS) are generally capable of maintaining proper performance when at nearly full capacity. However, the probability of performance problems is greatly increased when the filesystem becomes 100% full.
Unpredictability	It is difficult to predict precisely how a full filesystem may create issues because VMFS is a core element of the ESXi host. To ensure more efficient troubleshooting and system operation, keep VMFS below the thresholds defined above. The vSphere Client may report errors indicating an "Inaccessible" datastore or failures in writing VMFS metadata.

Note: This is not a comprehensive list of all possible issues, but it identifies important issues that have been reported in past instances of a full VMFS file system.

Environment

VMware vSphere ESXi 6.5.x
VMware vSphere ESXi 6.7.x
VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x
VMware vSphere ESX 9.x

Cause

Datastore exhaustion occurs when the total space consumed by virtual disks, snapshots, and swap files exceeds the physical capacity of the VMFS volume. ESXi requires a minimum of 100MB free for filesystem journals and metadata updates.

Resolution

What to do about a full VMware VMFS file system

When a VMFS is full, there are several courses of action:

Increase capacity: A VMFS file system may be stored on an external storage array. Depending on the features of the storage array, it may be possible to extend the external storage.

For more information on increasing datastore, see Increase VMFS Datastore Capacity and Unable to grow or expand a VMFS volume or datastore (1017662).

Storage vMotion virtual machines to other storage: If you cannot increase the storage available on the existing VMware VMFS datastore, but another datastore disk is accessible, it is possible to move virtual machines to the other datastore.

For more information on moving virtual machines, see Moving or copying a virtual machine within a VMware environment (1000936).

Add additional datastores: ESXi hosts are able to maintain up to 1024 datastores. If there is no existing free space where the virtual machine can be stored, additional storage may be added.
Reclaim space by removing snapshots: It is possible that additional space can be reclaimed by either committing or removing snapshots because old snapshots consume significant datastore space.

For more information, see the VMware Server Administration Guide. For more information on the topic of snapshots, see No more space for the redo log error when attempting to start a virtual machine and Overview of virtual machine snapshots in vSphere.

Clean up VMware VMFS: VMware strongly recommends that any files be backed up prior to being deleted from a VMFS datastore. Note: If the datastore is thin-provisioned, deleting files may not increase free space, as space reclamation on the array may not function when space usage is above a certain threshold.

Additional Information

This is an environmental condition. Configure monitoring systems to alert when datastore usage exceeds 80% to prevent capacity exhaustion.