Troubleshooting ESXi datastore or VMFS volume that is full or near capacity

Products

VMware vSphere ESXi

Issue/Introduction

Capacity Guidelines

ESX hosts store all files necessary for virtual machine operation in a VMware Virtual Machine File System (VMFS). Proper capacity management is critical because this file system is required for any operations related to a virtual machine. This document outlines potential issues related to a VMFS becoming full.

All VMFS (vmfs2, vmfs3, vmfs5,vmfs6) must retain a level of free space to ensure that the virtual machine functions properly. Ensure that these capacity thresholds are met:

200 Megabyte Capacity	VMware suggests 200 MB as the threshold where ESX system administrators take action to resolve capacity issues. VMFS volumes vary widely in size and it is impossible to define a precise percentage, but the risk of unexpected problems becomes greater at this point.
100 Megabyte Remaining	If left unchecked, a VMFS volume grows until only 100 MB of space remains, at which time issues may occur. 100 MB is defined as a critical threshold at which ESX system administrators must take immediate action to correct the issue because VMware VMFS stops growing at this point.

For more information, see Increase VMFS Datastore Capacity, and Unable to grow or expand a VMFS volume or datastore.

Potential Issues

This is a list of some potential issues that may affect virtual machines residing on a VMFS that is full:

Virtual machine startup failure	ESXi hosts are able to provide more RAM space than is available in the total physical RAM installed in the server. This is called Over Commitment. ESXi hosts create `.vswp` files for any virtual machine that has allocated memory greater than the physical RAM that is available to the virtual machine. The `<VMName>.vswp` file is created when the virtual machine is started. Without sufficient space, this file cannot be created, and the virtual machine is unable to start.
VMotion failures	ESXi hosts that are in a cluster of ESXi hosts that have VMware HA/DRS enabled may encounter failures when VMotion attempts to move a virtual machine from one host to another. It is possible for this failure to occur during off-hours when it may not be immediately identified as a failure because VMware DRS can be configured to VMotion virtual machines without interaction from an administrator.
Snapshot creation/commit	When a snapshot is taken or committed to the virtual machine, data is written to the VMFS. If the VMFS is unable to accommodate the additional data, the snapshot does not grow, does not get written, or is not committed. As a result, the virtual machine may fail.
Poor performance	File copy, `ls` commands and a variety of Linux-level functions perform poorly when the space is full because the VMware VMFS is a filesystem. Though less likely to affect virtual machine availability, these issues impact ESXi system administrators and must be resolved. Note: Journaling filesystems (like VMFS) are generally capable of maintaining proper performance when at nearly full capacity. However, the probability of performance problems is greatly increased when the filesystem becomes 100% full.
Unpredictability	It is difficult to predict precisely how a full filesystem may create issues because VMFS is a core element of the ESXi host. To ensure more efficient troubleshooting and system operation, keep VMFS below the thresholds defined above.

Note: This is not a comprehensive list of all possible issues, but it identifies important issues that have been reported in past instances of a full VMFS.

Environment

VMware vSphere ESXi 8.0.x
VMware vSphere ESXi 7.0.0

VMware vSphere ESXi 6.7

Resolution

What to do about a full VMware VMFS

When a VMFS is full, you have several courses of action:

Increase capacity: A VMFS may be stored on an external storage array. Depending on the features of the storage array, it may be possible to extend the external storage.

For more information on increasing datastore, see Increase VMFS Datastore Capacity and Unable to grow or expand a VMFS volume or datastore (1017662).

Storage vMotion virtual machines to other storage: If you cannot increase the storage available on the existing VMware VMFS, but another VMFS is accessible through that host, you can move virtual machines to the other VMware VMFS.

For more information on moving virtual machines, see Moving or copying a virtual machine within a VMware environment (1000936).

Add additional datastores: ESXi hosts are able to maintain up to 1024 datastores. If you are unable to identify existing free space where the virtual machine can be stored, you may add additional storage.
Remove old snapshots: It is possible that additional space can be reclaimed by either committing or removing snapshots because old snapshots also reside on VMware VMFS. This procedure requires direct knowledge of each virtual machine on the filesystem and must be performed by trained ESXi system administrators.

For more information, see the VMware Server Administration Guide. For more information on the topic of snapshots, see No more space for the redo log error when attempting to start a virtual machine and Overview of virtual machine snapshots in vSphere.

Clean up VMware VMFS: You can store files not related to the virtual machine operations because VMware VMFS is a filesystem. In this case, an experienced administrator carefully reviews the contents of the filesystem and determines what files are safe to remove. VMware strongly recommends that any files that are deleted from a VMware VMFS are first backed up, in case the file proves to be necessary.