Resolving "Datastore Full" Alerts Caused by Space Reclamation Issues.

Products

• VMware vCenter Server

Issue/Introduction

A VMware datastore reports a "Datastore Full" or low free space alert in vCenter, despite actual storage utilization on the Guest VM is being significantly lower (e.g., approximately 20%). This discrepancy can trigger unintended HA restarts for critical virtual machines, leading to service outages.

Symptoms:

• "Datastore Full" or "Low Free Space" alarm triggered in vCenter.
• Discrepancy between reported datastore usage (high) and actual disk utilization (low) on the RHEL VM.
• Virtual machines on the affected datastore may become inaccessible or restart unexpectedly due to HA events.
• Issue may be observed on the RHEL VMs where the fstrim commands are sent manually by the Guest/application layer.

Environment

VMware ESXi 8.x
VMware ESXi 9.x

Cause

The datastore reports stale space as "in-use" because the background space reclamation (UNMAP) operations are not effectively communicating free blocks to the storage array, or the storage array is unable to process these commands at the current rate.

Resolution

1. Immediate Manual Space Reclamation: When an immediate "Datastore Full" alert occurs and requires urgent resolution:

1. Establish an SSH session to the affected ESXi host.
2. Identify the Datastore UUID: esxcli storage vmfs extent list
3. Execute the manual unmap command to reclaim space: esxcli storage vmfs unmap -u <UUID> (Note: This forces immediate reclamation and usually clears the erroneous "Datastore Full" alert.)

2. Configuring Ongoing Maintenance (Throttling): To prevent the recurrence of this issue, configure automatic space reclamation at a fixed rate:

1. In the vSphere Client, navigate to the affected Datastore.
2. Select Configure > General > Space Reclamation > Edit.
3. Select Enable automatic space reclamation at fixed rate.
4. Specify an appropriate reclamation bandwidth (e.g., 10MB/s) to reduce the load on the storage array.
5. Click OK to save.
6. Unmount and remount the datastore to ensure the new settings take effect.

Best Practices:

• Guest OS: Configure Guest OS-level unmap/fstrim commands to ensure continuous, automated space reclamation from the guest layer.
• Monitoring: Use vSphere monitoring tools or esxcli to track the status of space reclamation operations.

Additional Information

For details on monitoring automatic UNMAP I/O, refer to: 

Monitor automatic unmap I/O issued by ESXi

How to throttle the unmap requests on Datastore ( Space Reclamation )  

If performance issues persist, please review your storage array capabilities or contact Broadcom Support: Contact Broadcom Support