ESXi 6.5 using VMFS-6 error: Insufficient disk space on datastore

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• This solution can also be used to address the problem before it starts complaining about space on the datastore. In this case, it would only have the symptom of not reclaiming the space on the array, and the following messages in vmkernel.log:

2017-10-05T01:56:38.713Z cpu3:66494)FSDisk: 300: Issue of delete blocks failed [sync:0] numDescs 1 Address is not aligned on required boundary 0x439540c9fb40

• Creating or consolidating snapshots during Storage vMotion on ESXi 6.5 host using VMFS-6 fails with the error:
  
  Insufficient disk space on datastore
   
• The datastore has sufficient free space.
• In the /var/log/vpxa.log file, you see entries similar to:
  
  -->          message = "An error occurred while taking a snapshot: There is not enough space on the file system for the selected operation."
  ...
  -->    msg = "Insufficient disk space on datastore ''."
   
• In the /var/log/vmkernel.log file, you see entries similar to:
  
  2017-10-05T01:56:38.713Z cpu3:66494)FSDisk: 300: Issue of delete blocks failed [sync:0] numDescs 1 Address is not aligned on required boundary 0x439540c9fb40
  ...
  2017-10-05T02:10:04.623Z cpu6:2841554)Fil3: 8222: Max no space retries (10) exceeded for caller Fil3_FileIO (status 'No space left on device')
   
• One or more datastore partitions have a Start Sector that is not aligned with a multiple of 1024.
  
  For example:
  # esxcli storage core device partition list
  Device                                Partition  Start Sector  End Sector  Type  Size
  -------------------------------------------------------------------------------------
  naa.60030d90eb30aa06053e4105ddc052e0          1           128  4404019167    fb  2254857747968   MyBrokenDatastore       
  naa.60030d908cdc38043b69a7b3522416d9          1          2048  2726297567    fb  1395863305728   MyWorkingDatastore

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Resolution

This issue has been fixed in ESXi 6.5.0d, build-5310538, available at VMware Downloads. Upgrading to this version, or a newer version, will prevent new datastores from being created with the wrong start sector, but it will not fix existing datastores.

For long-term resolution, for those datastores that were already created with the wrong start sector (128), follow these steps:

1. Upgrade to ESXi 6.5.0d, build-5310538, or newer version of ESXi.
2. Create new datastores.
3. Migrate the VMs from the faulty datastores to the new datastores
4. Delete the faulty datastores.

After this is done, you should have no further troubles with automatic UNMAP, thus it will prevent the datastore disk space issue.

Workaround:
To work around this issue:

1. From an ESXi host command line, you can fill the free space in the file system with zeros, using the esxcli storage vmfs unmap -u command:
   
   For example: esxcli storage vmfs unmap -u 58e36bcd-396ca5a1-f37b-6c3be5b653b8

Note: Give the above task some time to complete, as the array will have some work to do, after the ESXi host command has completed the zero operation, and we cannot tell when the array gets done. Maybe wait until the next day. Then you can manually run the reclaim operation, mentioned in your storage array management software documentation, through the array management software.

Additional Information

Thin Provisioning Block Space Reclamation (VAAI UNMAP) does not work