Disabling hardware accelerated locking (ATS) in ESXi
search cancel

Disabling hardware accelerated locking (ATS) in ESXi

book

Article ID: 319797

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

This article provides steps to disable ATS usage at different levels.
 
In a shared storage environment, when multiple hosts access the same VMFS datastore, specific locking mechanisms are used. These locking mechanisms prevent multiple hosts from concurrently writing to the metadata and ensure that no data corruption occurs.

VMFS supports SCSI Reservations and Atomic Test Set (ATS) locking.

ATS-only is a VAAI primitive on VMFS-5 datastores and supported VAAI-enabled arrays which performs all locking operations on a LUN instead of using SCSI reservations. VAAI may need to be disabled if the storage array devices in the environment do not support the hardware acceleration functionality or are not responding correctly to VAAI primitives.
 
Notes:
  • VAAI commands offloaded are bi-directional hand shakes between the ESXi Host and the Storage Array. 
    The Host sends a VAAI Command Sense Code that is understood by the Array, this is because the Array falls under VAAI supported list with its updated firmware version.
    If the Array does not Support VAAI then disable the hardware accelerated locking (ATS) in ESXi
  • Virtual Machines running on the datastore should be powered off before disabling ATS.


Environment

VMware vSphere ESXi 8.0.x
VMware vSphere ESXi 7.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.5


Resolution

Note: If running a VMware vSAN environment or ATS-only VMFS volumes, it is important that Hardware Assisted Locking (ATS) is not disabled. Doing so will result in an outage as there will be no locking mechanism available to use.
 
ATS usage can be disabled at different levels:

 

Note: Depending on its configuration and the type of underlying storage, a VMFS datastore can exclusively use the atomic test and set locking mechanism (ATS-only), or use a combination of ATS and SCSI reservations (ATS+SCSI). Datastores that use the ATS+SCSI mechanism include VMFS5 datastores that were upgraded from VMFS3. In addition, new VMFS5 datastores on storage devices that do not support ATS use the ATS+SCSI mechanism.
 
To display information related to VMFS locking mechanisms, run the following command:

esxcli storage vmfs lockmode list

 

Disabling ATS Usage per ESXi host level

 

Disabling ATS for Non ATS-Only VMFS datastores


To disable ATS on Non ATS only volumes, run the following ESXCLI:

esxcli system settings advanced set --int-value 0 --option /VMFS3/HardwareAcceleratedLocking

Note:
  • This option is effective immediately and does not need any reboot or mount/unmount of volumes.
  • The options should be disabled on all ESXi hosts accessing the VMFS datastores to have a consistent locking mode to be used from all ESXi hosts. Note that the ESXi does not have this requirement but there can be some potential issue from the array side on handling SCSI reservations as well as ATS commands on the same LUN from multiple hosts. Consult Storage Array Vendor for further details.
Disabling ATS for ATS-Only VMFS datastores


To disable ATS on ATS only volumes, run the following ESXCLI:

esxcli system settings advanced set --int-value 0 --option /VMFS3/HardwareAcceleratedLocking

Notes:
  • This option though reflects the value immediately but is not effective unless the datastore is unmounted and remounted.
  • This option need to be reset on all the ESXi hosts in the cluster.
  • When complete, mount/unmount the datastores on all ESXi hosts one by one.
  • Whether performing manual or rolling reboots of the ESXi hosts to achieve the unmount/remount, do not stop in the middle.
  • If running a version of ESXi between vSphere55U2 and vsphere55p09, you also need to disable ATS usage in VMFS heartbeat separately using the following ESXCLI:

    esxcli system settings advanced set --int-value 0 --option /VMFS3/UseATSForHBOnVMFS5

Disabling ATS usage per VMFS datastore

 

Disabling ATS for Non ATS-Only VMFS datastores

There is no option/need to disable ATS on such datastore and turning off ATS support on ESXi host (side effect of disabling for all datastores on that host) as mentioned earlier will work.

Disabling ATS for ATS-Only VMFS datastores

To disable ATS for ATS-Only VMFS datastores::

  1. Unmount the VMFS datastore from all the ESXi hosts which are using it. For this, this may require stopping SIOC. For more information, see Troubleshooting Storage I/O Control (311249).
  2. Execute this command to disable the ATS usage on VMFS datastore:

    vmkfstools --configATSOnly 0 /vmfs/volumes/VMFS-volume-name

    Alternatively, starting with vSphere 6.0, use the following ESXCLI:

    esxcli storage vmfs lockmode set -s|--scsi -l|--volume-label= VMFS label -u|--volume-uuid= VMFS UUID.

    When you have disabled, your datastore will be similar to the one mentioned in the Disabling ATS Usage per ESXi host level section and you can disable the ATS support at ESXi host level on the fly.
     
  3. Mount back the datastore on all ESXi hosts.

Disabling ATS Usage only for VMFS heartbeats


Starting with vSphere55u2, VMFS uses ATS for updating its heartbeat compared to plain scsi writes earlier.

To disable ATS heartbeat, run the following ESXLI command:

esxcli system settings advanced set -i 0 -o /VMFS3/UseATSForHBOnVMFS5

Note: It is not recommended to disable ATS heartbeat unless you see this set logs in the vmkernel.log file:

ATS Miscompare detected between test and set HB images at offset XXX on vol YYY

For more information, see: ESXi host loses connectivity to a VMFS datastore with VAAI ATS (326437.)

Workaround:
Steps for disabling ATS
1. Migrate all VMs which are running on affected datastore to another for stopping I/Os
2. Execute command in Resolution section.
3. Unmount affected datastores from all ESXi hosts
4. Mount back the datastores on all ESXi hosts.

Additional Information

Notes:
  • ATS is a SCSI command which if supported is used to atomically updated the contents of a sector on the disk and is can be used for various use cases.
  • ATS in itself does not provide any locking but VMFS in ESXi which uses on-disk locking for metadata consistency and access synchronization uses ATS command to achieve locking semantics.
  • Starting with vSphere55u2, VMFS uses ATS for its heartbeat updates (every 3 seconds) as well compared to scsi writes in older release.
Troubleshooting Storage I/O Control
Disabling hardware accelerated locking (ATS) in ESXi

Powered by Wolken Software