Datastore fails to mount with "Failed to initialize VMFS distributed locking on volume...Not Supported" - due to missing claim rules

Products

• VMware vSphere ESXi

Issue/Introduction

• A datastore fails to mount on one of more ESXi hosts.
  
  
• The backing LUN is an ATS-only volume and the LUN supports ATS
  
  
• On the impacted ESXi hosts, /var/log/vmkernel.log reports that ATS initialization is failed due to being 'not supported':

vmkwarning: cpu127:2106306 opID=4cbc49a6)WARNING: HBX: 2447: ATS-Only VMFS volume '<datastoreName>' is not mounted. This host does not support ATS, or ATS initialization failed.
vmkwarning: cpu127:2106306 opID=4cbc49a6)WARNING: HBX: 2468: Failed to initialize VMFS distributed locking on volume ########-########-####-####-############: Not supported
vmkernel: cpu127:2106306 opID=4cbc49a6)Vol3: 4768: Failed to get object 28 type 1 uuid ########-########-####-####-############ FD 0 gen 0 :Not supported
vmkwarning: cpu127:2106306 opID=4cbc49a6)WARNING: Fil3: 1643: Failed to reserve volume f532 28 1 ######## ######## ######## ######## 0 0 0 0 0 0 0

Environment

VMware vSphere ESXi (all versions)

Cause

This occurs if

• the storage array does not natively support VAAI, but requires use of a VAAI plugin to leverage VAAI funcitonality (the plugin translates VAAI commands to a form the array can handle)
• the claim rules necessary to leverage the plugin for these specific devices are not configured on the host

Confirmation:

• On query of VPD page 0xb0 (176 decimal), the array returns that ATS is suppored on the device:
  
  vsish -e get /storage/scsifw/devices/naa.################################/inquiryVPD/176/data
[0]: 0
[1]: 176
[2]: 0
[3]: 60
[4]: 0
[5]: 1  <=== ATS supported
[6]: 0
  ...
  
  
  
• The device reports ATS unsupported for the device: 
  
  esxcli storage core device vaai status get -d naa.################################
naa.################################
   VAAI Plugin Name:
   ATS Status: unsupported
   Clone Status: unsupported
   Zero Status: unsupported
   Delete Status: unsupported
   Ex Clone Status: unsupported 
  
  
  
• On a host on which the datastore is mounted (with ATS initialized successfully) VAAI_Filter and VAAI rules are configured for the device, similar to: 

# esxcli storage core claimrule list (output simplified)
Rule Class  Rule   Class    Type       Plugin            Matches                                                             
----------  -----  -------  ---------  ----------------  -------------------------------------------
Filter      5001  runtime   device     VAAI_FILTER       device=naa.################################    
Filter      5001  file      device     VAAI_FILTER       device=naa.################################              
VAAI        5001  runtime   device     VMW_VAAIP_T10     device=naa.################################ 
VAAI        5001  file      device     VMW_VAAIP_T10     device=naa.################################                       

• On the host on which ATS locking fails to initialize, these rules are not configured. 

 

Resolution

• Confirm with storage vendor the rules to be added
  
  
• Add the rules as per: Managing Hardware Acceleration on Block Storage Devices, see in particular the section: Add Hardware Acceleration Claim Rules

Additional Information

Similar symptoms may be seen where the array no longer reports the devices as supporting ATS. See: Datastore mount operation fails with the error "This host does not support ATS, or ATS initialization"