This KB provides the troubleshoot elements needed to determine if you have this issue and will provide a workaround specifically to address VMFS (ATS-only) volumes on SAS Storage(Active/Active or Active/Passive) failed to automount when ESXi host is upgraded to 7.0 U2 and onwards.
Note: This KB can be useful to all four VAAI ops we support : ATS Locking, Clone, Zero and Delete. Here the focus is on Locking.
Symptoms:
- ESXi host recently updated or upgraded to 7.0 U2 and onwards.
- SAS is used for Storage provisioning VMFS.
- You can see the impacted datastore's backing device size.
- You can consult its partition table and metadata.
- VMFS ATS-only datastores are not mounted.
- You can see in vmkernel the following logs (names and timestamps may vary depending on your environment):
2021-03-11T12:45:18.375Z cpu5:2113297)WARNING: HBX: 2419: ATS-Only VMFS volume 'Datastore' is not mounted. This host does not support ATS, or ATS initialization failed.
2021-03-11T12:45:18.375Z cpu5:2113297)WARNING: HBX: 2439: Failed to initialize VMFS distributed locking on volume 5fb4c81e-c702312a-3133-9440c93e9d60: Not supported
2021-03-11T12:45:18.375Z cpu5:2113297)Vol3: 4339: Failed to get object 28 type 1 uuid 5fb4c81e-c702312a-3133-9440c93e9d60 FD 0 gen 0 :Not supported
2021-03-11T12:45:18.375Z cpu5:2113297)WARNING: Fil3: 1534: Failed to reserve volume f532 28 1 5fb4c81e c702312a 40943133 609d3ec9 0 0 0 0 0 0 0
2021-03-11T12:45:18.375Z cpu5:2113297)Vol3: 4339: Failed to get object 28 type 2 uuid 5fb4c81e-c702312a-3133-9440c93e9d60 FD 4 gen 1 :Not supported
2021-03-11T12:45:18.425Z cpu4:2113309)HBX: 1016: 'Datastore': HB at offset 3145728 - Setting pulse failed: Not supported:
2021-03-11T12:45:18.425Z cpu4:2113309) [HB state abcdef02 offset 3145728 gen 71 stampUS 3782277213 uuid 604a0231-493248f2-ec8c-9440c93e9d60 jrnl <FB 0> drv 24.82 lockImpl 4 ip 172.xx.xx.51]
2021-03-11T12:45:18.425Z cpu4:2113309)WARNING: FSAts: 1568: Denying reservation access on an ATS-only vol 'Datastore'
- When you look at the device on SSH using the command "esxcli storage core device list", you see the following:
naa.600c0ff000514ed010c6b45f01000000:
Display Name: Local HPE Disk (naa.600c0ff000514ed010c6b45f01000000)
Has Settable Display Name: false
Size: 4768368
Device Type: Direct-Access
Multipath Plugin: HPP
Devfs Path: /vmfs/devices/disks/naa.600c0ff000514ed010c6b45f01000000
Vendor: HPE
Model: MSA 2050 SAS
Revision: V270
SCSI Level: 6
Is Pseudo: false
Status: on
Is RDM Capable: true
Is Local: true
Is Removable: false
Is SSD: false
- The command "esxcli storage core device vaai status get" returns the following:
naa.600c0ff000514ed010c6b45f01000000:
VAAI Plugin Name:
ATS Status: unsupported
Clone Status: unsupported
Zero Status: unsupported
Delete Status: unsupported
- If you still have a host in 7.0U1 to compare you can see the following:
naa.600c0ff000514ed010c6b45f01000000:
Display Name: HPE Serial Attached SCSI Disk (naa.600c0ff000514ed010c6b45f01000000)
Has Settable Display Name: true
Size: 4768368
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/naa.600c0ff000514ed010c6b45f01000000
Vendor: HPE
Model: MSA 2050 SAS
Revision: V270
SCSI Level: 6
Is Pseudo: false
Status: on
Is RDM Capable: true
Is Local: false
Is Removable: false
Is SSD: false
naa.600c0ff000514ed010c6b45f01000000:
VAAI Plugin Name:
ATS Status: supported
Clone Status: supported
Zero Status: supported
Delete Status: supported