NFSv3 Unmap feature with VAAI-NAS plugin
search cancel

NFSv3 Unmap feature with VAAI-NAS plugin

book

Article ID: 401327

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • The article explains about the Guest OS Unmap with NFSv3 and VAAI-NAS plugin
  • The Guest OS Unmap feature is added to NFSv3 by extending the current VAAI-NAS infrastructure.  The following primitives were added in the VAAI-NAS infrastructure to facilitate Unmap:
    1. VIXDISKLIB_NASPLUGIN_PRIM_GET_UNMAP_GRANULARITY
      The function of the primitive is to get the unmap related parameters from the backend, since the number of unmap calls being submitted to the backend largely depends on these parameters
    2. VIXDISKLIB_NASPLUGIN_PRIM_UNMAP                                                                                                                    The function of this primitive is to send a coalesced list of Unmap requests from the Guest OS to the VAAI-NAS plugin

 

Environment

VMware vSphere ESXi 9.0

 

Resolution

Notes/Constraints:

  • Unmap is supported only on thin-provisioned disks, provided a vendor supplied VAAI-NAS plugin is installed on the host, and the plugin supports Unmap feature.
  • During a VM power on, the Guest VM issues a SCSI inquiry command to query the parameters related to Unmap operation. The queried parameters are filled as applicable by the VAAI-NAS plugin.
  • If a Guest VM doesn't detect Unmap support for thin-provisioned disks despite the underlying infrastructure supporting it, a VM reboot is required to re-query the support.
  • Unmap through VAAI-NAS happens only when there are no VM snapshots. When VM snapshots are present, the snapshot infrastructure handles Unmap commands.
  • Unmap is supported only on NFSv3 datastores in vSphere 9.0 release, NFSv4.1 Unmap support will be added in a future vSphere release.


Unmap tuning parameters

There are a few advance configuration options introduced to fine-tune the NFSv3 Unmap behavior. The following sections explains the purpose of the several tuning options,

Disclaimer: The values used in the commands are examples and not to be interpreted as recommendations.

  • Option - /NFS/UnmapEnabled 

This option is added to enable/disable NFS Unmap support. By default NFS Unmap support is disabled. To enable the support, the below advance configuration option needs to be set on all the ESXi hosts that requires Unmap:

# get the current status of the NFS Unmap feature
[root@esxi-host:~] esxcfg-advcfg -g /NFS/UnmapEnabled
Value of UnmapEnabled is 0
 
# enable the NFS Unmap feature
[root@esxi-host:~] esxcfg-advcfg -s 1 /NFS/UnmapEnabled
Value of UnmapEnabled is 1

  • Option - /NFS/UnmapBatchWindowTimeoutMs 
    This option is added to control the waiting time to coalesce Unmap requests. The Unmap requests for the same vDisk within this interval will be coalesced and sent to the the VAAI-NAS plugin.

# the values are in milli-seconds, max is 100ms, min is 2ms
[root@esxi-host:~] esxcfg-advcfg -g /NFS/UnmapBatchWindowTimeoutMs
Value of UnmapBatchWindowTimeoutMs is 5
 
[root@esxi-host:~] esxcfg-advcfg -s 10 /NFS/UnmapBatchWindowTimeoutMs
Value of UnmapBatchWindowTimeoutMs is 10

  • Option - /NFS/UnmapMaxQueueDepth 
    This option is added to control the maximum number of Unmap requests that can be queued per datastore. Each Unmap request may have more than one (offset, length) pairs due to coalescing.

[root@esxi-host:~] esxcfg-advcfg -g /NFS/UnmapMaxQueueDepth
Value of UnmapMaxQueueDepth is 32
 
[root@esxi-host:~] esxcfg-advcfg -s 16 /NFS/UnmapMaxQueueDepth
Value of UnmapMaxQueueDepth is 16

  • Option - /NFS/UnmapMaxThreads 
    This option is added to control the number of Unmap threads provisioned to run an a given ESXi host.

# maximum number of threads is 128
[root@esxi-host:~] esxcfg-advcfg -g /NFS/UnmapMaxThreads
Value of UnmapMaxThreads is 16
 
[root@esxi-host:~] esxcfg-advcfg -s 32 /NFS/UnmapMaxThreads
Value of UnmapMaxThreads is 32

Unmap Statistics

  • From the ESXi host, the run time Unmap statistics could be read.
  • The statistics would help with further fine-tuning of the Unmap related advance configuration options. The Unmap related statistics have been added to the existing vaaistats node. Some of the entries are trimmed in the example for brevity,

[root@esxi-host:~] vsish -e get /vmkModules/nfsclient/mnt/<nfsv3-volume-label>/vaaistats
Vaai statistics {
   fullCloneSuccess:0
   fullCloneFailure:0
   ...
   extStatsFailure:0
   unmapSuccess:0
   unmapFailure:0
   unmapCancelSuccess:0
   unmapResetSuccess:0
   unmapGranularitySuccess:0
   unmapGranularityFailure:0
   unmapGranularityCancelSuccess:0
   unmapGranularityCancelFailure:0
   fullCloneSuccessTime:0
   ...
   extStatsFailureTime:0
   unmapSuccessTime:0
   unmapFailureTime:0
   unmapGranularitySuccessTime:0
   unmapGranularityFailureTime:0
   unmapLatencyHist:The time (us) spent for an unmap request to complete from VAAI-NAS infrastructure {
      min:0
      max:0
      mean:0
      count:0
      Histogram Bucket Array: {
         0                  (<=                 63)
         0                  (<=                127)
         0                  (<=                255)
         0                  (<=                511)
         0                  (<=               1023)
         0                  (<=               2047)
         0                  (<=               4095)
         0                  (<=               8191)
         0                  (<=              16383)
         0                  (<=              32767)
         0                  (<=              65535)
         0                  (<=             131071)
         0                  (<=             262143)
         0                  (<=             524287)
         0                  (>              524287)
      }
   }
   unmapBatchingHist:The number of unmap requests batched into a single call to VAAI-NAS infrastrucutre {
      min:0
      max:0
      mean:0
      count:0
      Histogram Bucket Array: {
         0                  (<=                  1)
         0                  (<=                  3)
         0                  (<=                  7)
         0                  (<=                 15)
         0                  (<=                 31)
         0                  (>                  31)
      }
   }
}

  • The statistics
  • represent the following:
Statistics Field Description
unmapSuccess Total number of successful Unmap calls 
unmapFailure Total number of failed Unmap calls
unmapGranularitySuccess Total number of successful Granularity IOCTL calls
unmapGranularityFailure Total number of failed granularity IOCTL calls
unmapSuccessTime Cumulative time taken to execute successful Unmap calls
unmapFailureTime Cumulative time taken to execute failed Unmap calls
unmapGranularitySuccessTime Cumulative time taken to execute successful Unmap granularity calls
unmapGranularityFailureTime Cumulative time taken to execute failed Unmap granularity calls
unmapLatencyHist Histogram that captures the different range of latencies observed
unmapBatchingHist Histogram that captures the efficiency of batching algorithm
unmapCancelSuccess Number of Unmap requests cancelled successfully (cancels are issued by Guest OS)
unmapCancelSuccess umber of Unmap resets processed successfully (resets are issued by Guest OS)

 

Unmap Logging

  • The NFSv3 Unmap feature related logs could be found in two log files,
  1. /var/log/vmkernel.log for Unmap platform related logs
  2. /var/log/syslog.log for Unmap VAAI-NAS infrastructure and 3rd party VAAI-NAS plugin related logs

Additional Information

For partners who are developing VAAI-NAS plugin, Unmap and other VAAI-NAS primitives' related information could be found in NAS VAAI Development Kit Guide 

Powered by Wolken Software