Frequently Asked Questions for vStorage APIs for Array Integration

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides answers to frequently asked questions about vStorage APIs for Array Integration (VAAI).

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 5.x

Resolution

vStorage APIs for Array Integration is a feature introduced in ESXi that provides hardware acceleration functionality. It enables your host to offload specific virtual machine and storage management operations to compliant storage hardware. With the storage hardware assistance, your host performs these operations faster and consumes less CPU, memory, and storage fabric bandwidth.

What does VAAI require?

VAAI requires:

• Standard, Enterprise, Enterprise Plus or Remote Office Branch Office licensing for ESXi hosts.
• Storage arrays that support VAAI storage-based hardware acceleration.
  
  How do I know if my storage array supports VAAI?
  
  To determine if your storage array supports VAAI, see the Hardware Compatibility List or consult your storage vendor. To enable the hardware acceleration on the storage array, check with your storage vendor. Some storage arrays require explicit activation of hardware acceleration support.
  
  Note:

• In https://compatibilityguide.broadcom.com/  there are options under the Features Category and Features fields where arrays can be searched against to verify if they are VAAI-certified.

What are the fundamental operations used by VAAI and what are their corresponding Advanced Settings?

VAAI uses these fundamental operations:

• Atomic Test & Set (ATS), which is used during creation and locking of files on the VMFS volume
• Clone Blocks/Full Copy/XCOPY, which is used to copy or migrate data within the same physical array
• Zero Blocks/Write Same, which is used to zero-out disk regions
• Thin Provisioning in ESXi 5.x and later hosts, which allows the ESXi host to tell the array when the space previously occupied by a virtual machine (whether it is deleted or migrated to another datastore) can be reclaimed on thin provisioned LUNs.
• Block Delete in ESXi 5.x and later hosts, which allows for space to be reclaimed using the SCSI UNMAP feature.

The fundamental operations are controlled by these advanced settings:

Advanced Parameter name	Description
HardwareAcceleratedLocking	Atomic Test & Set (ATS), which is used during creation of files on the VMFS volume
HardwareAcceleratedMove	Clone Blocks/Full Copy/XCOPY, which is used to copy data
HardwareAcceleratedInit	Zero Blocks/Write Same, which is used to zero-out disk regions

Note: For Thin Provisioning, enabling/disabling occurs on the array and not on the ESXi host.

What improvements have been made to VAAI for ESXi 5.x?

To encourage customers to use larger and fewer datastores, support for Thin Provisioning VAAI primitive has been added in ESXi 5.0.

In ESXi 5.x, support for NAS Hardware Acceleration is included with support for these primitives:

• Full File Clone – Like the Full Copy VAAI primitive provided for block arrays, this Full File Clone primitive enables virtual disks to be cloned by the NAS device.
• Native Snapshot Support – Allows creation of virtual machine snapshots to be offloaded to the array.
• Extended Statistics – Enables visibility to space usage on NAS datastores and is useful for Thin Provisioning.
• Reserve Space – Enables creation of thick virtual disk files on NAS.
  
  Note: Previously, the only supported VMDK type that could be created on NAS was thin.

How do I know if VAAI is enabled?

To determine if VAAI is enabled using the vSphere Client:

1. In the vSphere Client inventory panel, click the host.
2. Click the Configuration tab, then click Advanced Settings under Software.
3. Check that these options are set to 1 (enabled):
   
   DataMover.HardwareAcceleratedMove
   DataMover.HardwareAcceleratedInit
   VMFS3.HardwareAcceleratedLocking
   
   Note: These options are enabled by default.

On ESXi 5.x hosts, to determine if VAAI is enabled, run these commands and check if Int Value is set to 1 (enabled):

# esxcli system settings advanced list -o /DataMover/HardwareAcceleratedMove
# esxcli system settings advanced list -o /DataMover/HardwareAcceleratedInit
# esxcli system settings advanced list -o /VMFS3/HardwareAcceleratedLocking

You see output similar to:

Path: /VMFS3/HardwareAcceleratedLocking
Type: integer
Int Value: 1 <-- set to 1 if enabled
Default Int Value: 1
Min Value: 0
Max Value: 1
String Value:
Default String Value:
Valid Characters:
Description: Enable hardware accelerated VMFS locking (requires compliant hardware)

Note: For Thin Provisioning, enabling/disabling occurs on the array and not on the ESXi host.


When enabling or disabling VAAI settings, is a reboot required?

No, a host reboot is not required. The changes made may or may not be effective immediately depending on use case. For details please see below table:.

Use Case	Effective	ESXi Host reboot Needed	Volume mount/unmount Needed	Remark
Disabling Hardware Accelerated Move (XCOPY) in ESXi	Immediate	No	No
Disabling Hardware Accelerated Init (WRITESAME) in ESXi	Immediate	No	No
Disabling ATS Usage at ESXi host level - For upgraded VMFS datastores a.k.a Non ATS-Only VMFS datastores	Immediate	No	No
Disabling ATS Usage at ESXi host level - for ATS-Only VMFS datastores	No	No	Yes	Not effective unless the datastore is unmounted and mounted back. If you reboot the ESXi host to achieve the mount/unmount you will potentially end up in a mixture i.e. ATS+SCSI locking till all the host are rebooted which is not recommended
Disabling ATS usage per VMFS datastore - For upgraded VMFS datastores a.k.a Non ATS-Only VMFS datastores	NA	NA	NA	There is no option/need to disable ATS on such datastore.
Disabling ATS usage per VMFS datastore - For ATS-Only VMFS datastores	No	No	Yes	Datastore must be unmounted before executing command to disable ATS usage.
Disabling ATS Usage only for VMFS heartbeats	Immediate	No	No

What happens if I have VAAI enabled on the host but some of my disk arrays do not support it?

When storage devices do not support or provide only partial support for the host operations, the host reverts to its native methods to perform the unsupported operations.

Are there any caveats that I should be aware of in ESXi 5.x?

Use of the Atomic Test-and-Set (ATS) primitive depends on the type of filesystem:

On VAAI Hardware	New VMFS-5	Upgraded VMFS-5	VMFS-3
Single-extent datastore	ATS only [1]	ATS, but fall back to SCSI-2 reservations	ATS, but fall back to SCSI-2 reservations
Multi-extent datastore	Only allow spanning on ATS hardware [2]	ATS, except when locks on non-head	ATS, except when locks on non-head

Notes:

• If a new VMFS-5 is created on a non-ATS storage device, SCSI-2 reservations will be used.
• When creating a multi-extent datastore where ATS is used, the vCenter Server will filter out non-ATS devices, so that only devices that support the ATS primitive can be used.

VAAI hardware offload cannot be used when:

• The source and destination VMFS volumes have different block sizes
• The source file type is RDM and the destination file type is non-RDM (regular file)
• The source VMDK type is eagerzeroedthick and the destination VMDK type is thin
• The source or destination VMDK is any kind of sparse or hosted format
• Cloning a virtual machine that has snapshots because this process involves consolidating the snapshots into the virtual disks of the target virtual machine.
• The logical address and/or transfer length in the requested operation is not aligned to the minimum alignment required by the storage device (all datastores created with the vSphere Client are aligned automatically)
• The VMFS datastore has multiple LUNs/extents spread across different arrays

Notes:

• When using enhanced vMotion (move both VM and datastore simultaneously), the source and destination hosts must both see the source and destination datastores, or the file transfer defaults to NFC over the the management network, instead of VAAI.
• Hardware cloning between arrays (even if within the same VMFS datastore) does not work.

What do I need to know about the Hardware Acceleration Support Status?

If you go to Host > Configuration > Storage, you can see the Hardware Acceleration Status in the panel on the right side.

For each storage device and datastore, the vSphere Client displays the hardware acceleration support status in the Hardware Acceleration column of the Devices view and the Datastores view.

The status values are Unknown, Supported, and Not Supported. The initial value is Unknown. The status changes to Supported after the host successfully performs the offload basic operations. If the offload operation fails, the status changes to Not Supported.

To determine if your storage device supports VAAI, test the Full Copy VAAI primitive:

1. Using the vSphere Client, browse the datastore and locate a virtual disk (VMDK) of at least 4 MB that is not in use.
2. Copy the virtual disk to a new file.
3. Check the Hardware Acceleration status to verify that it changes from Unknown to either Supported or Not Supported.

Note: VAAI primitives can also be tested by creating a virtual machine with at least one new virtual disk, or cloning a virtual machine.


Can I check the VAAI status from the command line?

• On ESXi 5.x
  
  To check the VAAI status, run this command:
  
  # esxcli storage core device vaai status get
  
  You see output similar to:
  
  mpx.vmhba1:C0:T1:L0
  VAAI Plugin Name:
  ATS Status: unsupported
  Clone Status: unsupported
  Zero Status: unsupported
  Delete Status: unsupported
  
  mpx.vmhba1:C0:T0:L0
  VAAI Plugin Name:
  ATS Status: unsupported
  Clone Status: unsupported
  Zero Status: unsupported
  Delete Status: unsupported
  
  

Are changes to VAAI settings logged?

Changes to VAAI advanced settings (for example, enabling or disabling VAAI) are recorded in the VMkernel logs at /var/log/vmkernel (for ESXi 5.0 vmkernel.log) or /var/log/messages.

After enabling VAAI using the advanced settings, the VMkernel log contains entries similar to:

cpuN:1234)Config: 297: "HardwareAcceleratedMove" = 1, Old Value: 0, (Status: 0x0)
cpuN:1234)Config: 297: "HardwareAcceleratedInit" = 1, Old Value: 0, (Status: 0x0)
cpuN:1234)Config: 297: "HardwareAcceleratedLocking" = 1, Old Value: 0, (Status: 0x0)

After disabling VAAI using the advanced settings, the VMkernel log contains entries similar to:

cpuN:1234)Config: 297: "HardwareAcceleratedMove" = 0, Old Value: 1, (Status: 0x0)
cpuN:1234)Config: 297: "HardwareAcceleratedInit" = 0, Old Value: 1, (Status: 0x0)
cpuN:1234)Config: 297: "HardwareAcceleratedLocking" = 0, Old Value: 1, (Status: 0x0)

 

 

VAAI For NAS Storage

In ESXi 5.x, NAS Hardware Acceleration was introduced with support for these primitives:

· Full File Clone – Like the Full Copy VAAI primitive provided for block arrays, this Full File Clone primitive enables virtual disks to be cloned by the NAS device.

· Native Snapshot Support – Allows creation of virtual machine snapshots to be offloaded to the array.

· Extended Statistics – Enables visibility to space usage on NAS datastores and is useful for Thin Provisioning.

· Reserve Space – Enables creation of thick virtual disk files on NAS.

With NAS storage devices, the hardware acceleration integration is implemented through vendor-specific NAS plug-ins. These plug-ins are typically created by vendors and are distributed as VIB packages through a website. No claim rules are required for the NAS plug-ins to function.

 

Installing the NAS Plugin

1. Place the host into maintenance mode

 

2. Set the host acceptance level:

# esxcli software acceptance set–-level=value

The host acceptance level must be the same or less restrictive than the acceptance level of any VIB you want to add to the host. The value can be one of the following:

· VMwareCertified

· VMwareAccepted

· PartnerSupported

· CommunitySupported

3. Install the VIB package:

# esxcli software vib install –-viburl=URL

The URL specifies the URL to the VIB package to install. http:, https:, ftp:, and file: are supported.

4. Restart the host for the installation to take effect

 

Verify Hardware Acceleration Status for NAS

You can verify the NAS hardware acceleration status using the following command:

# esxcli storage nfs list

The Hardware Acceleration column in the output shows the status

 

Disabling VAAI when using NAS storage

 

To disable NAS VAAI functionality the plugin must be removed using the below commands.

1. Log into ESXi using SSH/DCUI and execute command:

# esxcli software vib remove –-vibname=name
For Example: # esxcli software vib remove –-vibname=NetAppNasPlugin

2. Verify that the plugin has been removed using command:

esxcli software vib list

3. Reboot the ESXi host to take effect.

 

Note: The advanced configuration options for Block VAAI mentioned above (DataMover.HardwareAcceleratedMove, DataMover.HardwareAcceleratedInit, VMFS3.HardwareAcceleratedLocking) do not affect NAS VAAI functionality

Additional Information

This is a list of commonly used SCSI opcodes related to VAAI operations:

0x93	WRITE SAME(16)
0x41	WRITE SAME(10)
0x42	UNMAP
0x89	SCSI COMPARE and WRITE - ATS
0x83	EXTENDED COPY

Occasionally, failures during VAAI operations would be logged in the vmkernel.log file under /var/log as shown in this example:

Cmd(0x41248092e240) 0x42, CmdSN 0x13bb23 to dev "naa.xxxxxxxx" failed H:0x0 D:0x2