Monitor automatic unmap I/O issued by ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

With VMFS-6, space reclamation is now an automatic, but asynchronous process. This means that the space will not be reclaimed immediately but over time and you might find your self needing to monitor this process.  We will go over the following topics related to Automatic Umap for VMFS-6. 

How to check the automatic unmap I/O per VMFS6 datastore/volume issued by an ESXi host.and check if automatic unmap I/O has been issued by an ESXi host. 

How to monitor the Umap activity via ESX top. 

Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x 

Resolution

How to check the automatic unmap I/O per VMFS6 datastore/volume issued by an ESXi host.and check if automatic unmap I/O has been issued by an ESXi host. 

• On an SSH session to an ESXi host, access the VSI shell prompt by running:

vsish

• Then navigate as follows:
  
  /> cd /vmkModules/vmfs3/auto_unmap/volumes/

 

• To list the VMFS6 datastores/volumes for which unmap is being issued, run:

/> ls

 

• To view the automatic unmap stats for a specific datastore:

/> get /vmkModules/vmfs3/auto_unmap/volumes/<datastoreName>/properties

 

• Alternatively, directly run:

vsish -e get /vmkModules/vmfs3/auto_unmap/volumes/<datastoreName>/properties

 

Sample output:

Volume specific unmap information {
   Volume Name                  :<datastoreName>
   Volume: Num SchedQs          :1
   FS Major Version             :24
   Metadata Alignment           :4096
   Allocation Unit/Blocksize    :1048576
   Unmap granularity in File    :1048576
   Volume: Unmap IOs            :1446
   Volume: Unmapped blocks      :85135
   Volume: Num wait cycles      :1
   Volume: Num from scanning    :3948
   Volume: Num from heap pool   :391
   Volume: Total num cycles     :961963
   Unmaps processed in last minute:0
   Volume: Unmap SchedQ IOs     :1446
}

Note: Statistics are reset on ESXi host reboot.

 

How to monitor the Umap activity via ESX top. 

From the following pure article  Monitoring Automatic VMFS-6 UNMAP in ESXi 

SSH into the ESXi host, type “esxtop” then enter and then type “u” then “f” then select VAAI statistics with “o”.

On the far right, you will see delete stats, aka UNMAP:

DELETE is a counter for UNMAP commands issued to that device, from that host since the last reboot. DELETE_F is a counter for failed UNMAP commands issued to that device, from that host since the last reboot. Note that this isn’t like how many times someone ran the UNMAP command. It is literally a count of UNMAP I/Os. A single UNMAP operation could include thousands of UNMAP commands.

The last column is not a counter, but instead a indicator of the throughput of any UNMAP operations going on right now from that host to that device. So if no UNMAP operations are running right now, it will be zero.

Additional Information

Non-zero values for Volume: Unmap IOs and Volume: Unmapped Blocks indicate that automatic unmap I/Os have been issue by a host for a given VMFS6 volume since last reboot. 

 

Check the output at intervals (e.g. daily) for incrementing values of  Volume: Unmap IOs and Volume: Unmapped Blocks to confirm ongoing issue of automatic unmap I/Os to the volume on the host. 

 

Unmap metrics can also be monitored via the VAAI stats on esxtop - see  Monitoring VMware vSphere Storage APIs – Array Integration using esxtop and maintaining the status as enabled. Note, however, that esxtop unmap metrics will include any manual unmap IOs issued, as well as automatic unmap IOs.