VMFS (ATS-only) volumes on SAS Storage failed to automount when upgraded to 7.0 U2 and onwards
Article ID: 339566
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
This KB provides the troubleshoot elements needed to determine if you have this issue and will provide a workaround specifically to address VMFS (ATS-only) volumes on SAS Storage(Active/Active or Active/Passive) failed to automount when ESXi host is upgraded to 7.0 U2 and onwards.
Symptoms:
Note: This KB can be useful to all four VAAI ops we support : ATS Locking, Clone, Zero and Delete. Here the focus is on Locking.
Symptoms:
- ESXi host recently updated or upgraded to 7.0 U2 and onwards.
- SAS is used for Storage provisioning VMFS.
- You can see the impacted datastore's backing device size.
- You can consult its partition table and metadata.
- VMFS ATS-only datastores are not mounted.
- You can see in vmkernel the following logs (names and timestamps may vary depending on your environment):
2021-03-11T12:45:18.375Z cpu5:2113297)WARNING: HBX: 2419: ATS-Only VMFS volume 'Datastore' is not mounted. This host does not support ATS, or ATS initialization failed.
2021-03-11T12:45:18.375Z cpu5:2113297)WARNING: HBX: 2439: Failed to initialize VMFS distributed locking on volume 5fb4c81e-c702312a-3133-9440c93e9d60: Not supported
2021-03-11T12:45:18.375Z cpu5:2113297)Vol3: 4339: Failed to get object 28 type 1 uuid 5fb4c81e-c702312a-3133-9440c93e9d60 FD 0 gen 0 :Not supported
2021-03-11T12:45:18.375Z cpu5:2113297)WARNING: Fil3: 1534: Failed to reserve volume f532 28 1 5fb4c81e c702312a 40943133 609d3ec9 0 0 0 0 0 0 0
2021-03-11T12:45:18.375Z cpu5:2113297)Vol3: 4339: Failed to get object 28 type 2 uuid 5fb4c81e-c702312a-3133-9440c93e9d60 FD 4 gen 1 :Not supported
2021-03-11T12:45:18.425Z cpu4:2113309)HBX: 1016: 'Datastore': HB at offset 3145728 - Setting pulse failed: Not supported:
2021-03-11T12:45:18.425Z cpu4:2113309) [HB state abcdef02 offset 3145728 gen 71 stampUS 3782277213 uuid 604a0231-493248f2-ec8c-9440c93e9d60 jrnl <FB 0> drv 24.82 lockImpl 4 ip 172.xx.xx.51]
2021-03-11T12:45:18.425Z cpu4:2113309)WARNING: FSAts: 1568: Denying reservation access on an ATS-only vol 'Datastore'
2021-03-11T12:45:18.375Z cpu5:2113297)WARNING: HBX: 2439: Failed to initialize VMFS distributed locking on volume 5fb4c81e-c702312a-3133-9440c93e9d60: Not supported
2021-03-11T12:45:18.375Z cpu5:2113297)Vol3: 4339: Failed to get object 28 type 1 uuid 5fb4c81e-c702312a-3133-9440c93e9d60 FD 0 gen 0 :Not supported
2021-03-11T12:45:18.375Z cpu5:2113297)WARNING: Fil3: 1534: Failed to reserve volume f532 28 1 5fb4c81e c702312a 40943133 609d3ec9 0 0 0 0 0 0 0
2021-03-11T12:45:18.375Z cpu5:2113297)Vol3: 4339: Failed to get object 28 type 2 uuid 5fb4c81e-c702312a-3133-9440c93e9d60 FD 4 gen 1 :Not supported
2021-03-11T12:45:18.425Z cpu4:2113309)HBX: 1016: 'Datastore': HB at offset 3145728 - Setting pulse failed: Not supported:
2021-03-11T12:45:18.425Z cpu4:2113309) [HB state abcdef02 offset 3145728 gen 71 stampUS 3782277213 uuid 604a0231-493248f2-ec8c-9440c93e9d60 jrnl <FB 0> drv 24.82 lockImpl 4 ip 172.xx.xx.51]
2021-03-11T12:45:18.425Z cpu4:2113309)WARNING: FSAts: 1568: Denying reservation access on an ATS-only vol 'Datastore'
- When you look at the device on SSH using the command "esxcli storage core device list", you see the following:
naa.600c0ff000514ed010c6b45f01000000:
Display Name: Local HPE Disk (naa.600c0ff000514ed010c6b45f01000000)
Has Settable Display Name: false
Size: 4768368
Device Type: Direct-Access
Multipath Plugin: HPP
Devfs Path: /vmfs/devices/disks/naa.600c0ff000514ed010c6b45f01000000
Vendor: HPE
Model: MSA 2050 SAS
Revision: V270
SCSI Level: 6
Is Pseudo: false
Status: on
Is RDM Capable: true
Is Local: true
Is Removable: false
Is SSD: false
Display Name: Local HPE Disk (naa.600c0ff000514ed010c6b45f01000000)
Has Settable Display Name: false
Size: 4768368
Device Type: Direct-Access
Multipath Plugin: HPP
Devfs Path: /vmfs/devices/disks/naa.600c0ff000514ed010c6b45f01000000
Vendor: HPE
Model: MSA 2050 SAS
Revision: V270
SCSI Level: 6
Is Pseudo: false
Status: on
Is RDM Capable: true
Is Local: true
Is Removable: false
Is SSD: false
- The command "esxcli storage core device vaai status get" returns the following:
naa.600c0ff000514ed010c6b45f01000000:
VAAI Plugin Name:
ATS Status: unsupported
Clone Status: unsupported
Zero Status: unsupported
Delete Status: unsupported
VAAI Plugin Name:
ATS Status: unsupported
Clone Status: unsupported
Zero Status: unsupported
Delete Status: unsupported
- If you still have a host in 7.0U1 to compare you can see the following:
naa.600c0ff000514ed010c6b45f01000000:
Display Name: HPE Serial Attached SCSI Disk (naa.600c0ff000514ed010c6b45f01000000)
Has Settable Display Name: true
Size: 4768368
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/naa.600c0ff000514ed010c6b45f01000000
Vendor: HPE
Model: MSA 2050 SAS
Revision: V270
SCSI Level: 6
Is Pseudo: false
Status: on
Is RDM Capable: true
Is Local: false
Is Removable: false
Is SSD: false
naa.600c0ff000514ed010c6b45f01000000:
VAAI Plugin Name:
ATS Status: supported
Clone Status: supported
Zero Status: supported
Delete Status: supported
Display Name: HPE Serial Attached SCSI Disk (naa.600c0ff000514ed010c6b45f01000000)
Has Settable Display Name: true
Size: 4768368
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/naa.600c0ff000514ed010c6b45f01000000
Vendor: HPE
Model: MSA 2050 SAS
Revision: V270
SCSI Level: 6
Is Pseudo: false
Status: on
Is RDM Capable: true
Is Local: false
Is Removable: false
Is SSD: false
naa.600c0ff000514ed010c6b45f01000000:
VAAI Plugin Name:
ATS Status: supported
Clone Status: supported
Zero Status: supported
Delete Status: supported
Environment
VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 7.0.0
Cause
Until 7.0 U2, VMWare default MPP plugin NMP claimed SAS devices. In 7.0 U2, the default MPP for SAS devices was changed to HPP. Like NMP, HPP also cannot differentiate between local/remote SAS targets unless in case of ALUA targets. In case of NMP also user would have added a NMP SATP claim rule forcing the device to be claimed by VMW_SATP_DEFAULT_AA/VMW_SATP_DEFAULT_AP which will in turn mark the device remote. We need a similar setting with HPP as well which user will have to explicitly setup for the device to mark it as remote. Because this setting is not done, the device is marked as local and hence VAAI operations will be disabled as VMWare disables VAAI operations on local HDDs.
Resolution
Currently there is no resolution.
While we are working on the configuration option with HPP , the current plan is to use NMP itself for the remote SAS target. To workaround the issue follow the below steps.
Workaround:
Check whether the devices with the above vendor model are claimed by NMP.
While we are working on the configuration option with HPP , the current plan is to use NMP itself for the remote SAS target. To workaround the issue follow the below steps.
Workaround:
Case 1: If ESXi is already updated to 7.0 U2 and onwards, add the claimrule using the steps below and then do an unclaim and reclaim of the devices, and if that doesn’t work do a host reboot.
Case 2: If planning to upgrade to 7.0 U2 and onwards, prior to upgrade add the claimrule using the steps below to overcome this issue.
Steps to Workaround:
Steps to Workaround:
- Make a note of the Vendor/Model string of the device using the command as below. If multiple devices from same target vendor/model, the setting will apply to all devices.
[root@******:~] esxcli storage core device list -d <device naa/mpx/t10 ID>
Example output:
naa.600c0ff0005150c90ec6b45f01000000:
Display Name: Local HPE Disk
(naa.600c0ff0005150c90ec6b45f01000000)
Has Settable Display Name: false
Size: 4768368
Device Type: Direct-Access
Multipath Plugin: HPP
Devfs Path: /vmfs/devices/disks/naa.600c0ff0005150c90ec6b45f01000000
Vendor: HPE
Model: MSA 2050 SAS
- Add the claim rule (rule number below 50)
[root@******:~] esxcli storage core claimrule add -r 49 -t vendor -V <Your device vendor> -M <Your device model> -P NMP --force-reserved
- Load the claim rule
[root@******:~] esxcli storage core claimrule load
Check claim rule list to confirm claim rule added successfully
[root@******:~] esxcli storage core claimrule list
Size: 4768368
Device Type: Direct-Access
Multipath Plugin: HPP
Devfs Path: /vmfs/devices/disks/naa.600c0ff0005150c90ec6b45f01000000
Vendor: HPE
Model: MSA 2050 SAS
- Add the claim rule (rule number below 50)
[root@******:~] esxcli storage core claimrule add -r 49 -t vendor -V <Your device vendor> -M <Your device model> -P NMP --force-reserved
- Load the claim rule
[root@******:~] esxcli storage core claimrule load
Check claim rule list to confirm claim rule added successfully
[root@******:~] esxcli storage core claimrule list
- For case 1 do the an unclaim and and reclaim of the devices, and if that doesn’t help do a ESXi host reboot.
- For case 2 after setting the claimrule go ahead with the ESXi host upgrade.
Check whether the devices with the above vendor model are claimed by NMP.
Additional Information
Reverting to a previous version of ESXi https://knowledge.broadcom.com/external/article/316592
Disabling hardware accelerated locking https://knowledge.broadcom.com/external/article/319797
Disabling hardware accelerated locking https://knowledge.broadcom.com/external/article/319797
Feedback
Yes
No
Powered by
