ESXi hosts fail to mount VMFS5 volumes that are formatted with ATS-only capabilities

Products

VMware Live Recovery VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Symptoms:

If a virtual machine is registered and the ESX host is rebooted, the virtual machine becomes inaccessible (orphaned) after reboot.
If a virtual machine is configured for Fault Tolerance and if the primary ESX host is restarted, the secondary virtual machine will not be created since the VMFS5 volume may fail to mount on the original primary server after restart.
If virtual machines are configured for auto-start after power on/reboot, the virtual machines fail to start as the VMFS5 volume mounts.
If a VMFS5 volume is unmounted and its device is detached, removed from an existing ESX server, and then presented to a new server (or presented back to the same server again), it fails to mount the VMFS5 volume.
In vCenter Server, you see an error similar to:

A general system error occurred: Timed-out while waiting to get datastore information from host
In the vmkernel.log file of the ESXi 5.0 host that has issues mounting the VMFS5 volume, you see entries similar to:

Setting pulse [HB state abcdef02 offset 3354624 gen 1 stampUS 12741765582 uuid 4e6f910f-9f0cbd23-bb3d-14feb5c9a0ec jrnl <FB 0> drv 14.54] on vol 'jrb163_52' failed: Not supported
YYYY-MM-DD TIME 325Z cpu30:5872)WARNING: FSAts: 1263: Denying reservation access on an ATS-only vol 'jrb163_52'
YYYY-MM-DD TIME 325Z cpu30:5872)WARNING: HBX: 1889: Failed to initialize VMFS3 distributed locking on volume 4e6e0c49-a955df7d-b6a3-14feb5c9aa1d: Not supported
YYYY-MM-DD TIME 325Z cpu30:5872)FSS: 890: Failed to get object f530 28 1 4e6e0c49 a955df7d fe14b6a3 1daac9b5 0 0 0 0 0 0 0 :Not supported
YYYY-MM-DD TIME 325Z cpu30:5872)WARNING: Fil3: 2032: Failed to reserve volume f530 28 1 4e6e0c49 a955df7d fe14b6a3 1daac9b5 0 0 0 0 0 0 0
YYYY-MM-DD TIME 325Z cpu30:5872)FSS: 890: Failed to get object f530 28 2 4e6e0c49 a955df7d fe14b6a3 1daac9b5 4 1 0 0 0 0 0 :Not supported

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
Attempting to perform Test Failover on Site Recovery Manager (SRM) 5.0 fails.
The failover host may not mount the replicated VMFS5 volume correctly and reports this error:

Error - Failed to recover datastore 'DatastoreName'. VMFS volume residing on recovered devices "DeviceNumber" and expected to be auto-mounted during HBA rescan cannot be found
In the SRM 5.0 vmware-dr.log file, you see the error similar to:

Failed to recover datastore 'DatastoreName'. VMFS volume residing on recovered devices '"DeviceNumber"' and expected to be auto-mounted during HBA rescan cannot be found.

Environment

VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.0
VMware vCenter Server 5.1.x
VMware vCenter Server 5.5.x
VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.1
VMware vCenter Server 5.0.x
VMware vCenter Site Recovery Manager 5.0.x

Cause

When a VMFS-5 volume formatted on a VAAI capable device, it uses ATS locking, exclusively termed as ATS only. To check if a VMFS volume is configured with ATS only, use this command:

# vmkfstools -Ph -v1 /vmfs/volumes/VMFS-volume-name

You see output similar to:

VMFS-5.54 file system spanning 1 partitions.
File system label (if any): ats-test-1
Mode: public ATS-only

When an ESX host discovers and registers new and existing devices, the VAAI plug-in module is loaded if the device is deemed to be VAAI capable during PSA device registration. In this particular case, ESX may fail to register device correctly with VAAI plug-in module as you are not reading the capacity of the volume correctly to apply the equivalent VAAI Plugin module. As a result, the VMFS5 volume may fail to mount.

This issue is seen to occur in EMC arrays Clariion CX4, NS, and VNX arrays, but is not restricted to these.

Resolution

This issue is resolved in ESXi 5.0 Update 1, available at VMware Downloads. For more information about this version, see the VMware ESXi 5.0 Update 1 Release Notes.

This issue is also resolved in ESXi 5.1 (See mentions of VAAI in 5.1, 5.1 U1, and 5.1U2 Release Notes in 5.1 Documentation Library for exception.)

To work around this issue in ESXi 5.0, disable VAAI ATS locking mechanism on your hosts. For more information, see Disabling the VAAI functionality in ESXi/ESX (1033665)

Additional Information

Disabling VAAI entirely on the ESXi host may introduce issues in the environment. Instead of disabling VAAI for all devices, you can disable it only for the affected LUN without impacting other LUNs.
- To downgrade VMFS-5 from ATS capability, run this command:
  
  vmkfstools --configATSOnly 0 device path
- To upgrade VMFS-5 to ATS capability, run this command:
  
  vmkfstools --configATSOnly 1 device path
  
  Where device path is the path of the head extent device on which VMFS-5 was formatted.
  
  For example:
  
  ~ # vmkfstools --configATSOnly 0 /vmfs/devices/disks/naa.600601607ae02600e85affc3f022e411:1
  
  VMware ESX Question:
  VMFS on device /dev/disks/naa.600601607ae02600e85affc3f022e411:1 will be upgraded to or downgraded from ATS capability. Please ensure that the VMFS-5 volume is not in active use by any local or remote ESX 4.x servers.
  
  Continue with configuration of ATS capability?
  
  0) _Yes
  1) _No
  
  Select a number from 0-1: 0
  
  Checking if remote hosts are using this device as a valid file system. This may take a few seconds...
  Downgrading VMFS-5 on '/dev/disks/naa.600601607ae02600e85affc3f022e411:1' from ATS capability...done

In ESXi 5.0 and later, the HP P2000 MSA iSCSI/FC Array requires the VIB hp_vaaip_p2000_offline-bundle-210.zip to be installed. HP does not include the VIB on their vSphere ESXi 5.x install media. If you have installed ESXi from an older media, it may still have the VIB and therefore VAAI hardware acceleration. The re-installed hosts lacking the VIB show entries similar to:

YYYY-MM-DD TIME 519Z [65AE2B70 info 'Vimsvc.ha-eventmgr'] Event 2467 : ATS-Only VMFS volume 'HP 2000 SAN Vol1' not mounted. Host does not support ATS or ATS initialization has failed.

To resolve this issue, install the hp_vaaip_p2000_offline-bundle-210.zip after the upgrade. To download hp_vaaip_p2000_offline-bundle-210.zip, see the HP depot.

Note: The preceding link was correct as of 21 January 2015. If you find the link is broken, provide a feedback and a VMware employee will update the link.

This issue can also occur on ESXi 5.5 hosts installed from a HP custom image, this install media lacks the VAAI plugin required to communicate with the LUNs using VAAI primitives. Resolve this by installing the HP VAAI plug-in manually. For more information, see:

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box.
Performing a rescan of the storage on an ESX/ESXi host
Identifying disks when working with VMware ESXi/ESX
Frequently Asked Questions for vStorage APIs for Array Integration
EMC CX and VNX Firmware and ESX requirements for vStorage APIs for Array Integration (VAAI) support
VMware ESXi 5.0, Patch ESXi50-Update01: VMware ESXi 5.0 Complete Update 1
ESXi ホストが ATS 専用（ATS のみ）機能でフォーマットされた VMFS5 ボリュームのマウントに失敗する
ESXi 主机无法挂载使用 ATS-only 功能进行格式化的 VMFS5 卷