VMFS datastore creation fails for NVMe luns.
Article ID: 374573
Updated On:
Products
Issue/Introduction
Symptoms:
A lun presented using NVMe controller faces issue with mounting.
vmkernel.log
2024-08-12T16:38:47.730Z cpu39:2100731 opID=16500354)World: 12077: VC opID lzfrf27w-51793-auto-13yr-h5:70015662-56-d1-4cd0 maps to vmkernel opID 16500354
2024-08-12T16:38:47.730Z cpu39:2100731 opID=16500354)LVM: 13505: LVMProbeDevice failed with status "Device does not contain a logical volume".
2024-08-12T16:38:47.732Z cpu39:2100731 opID=16500354)WARNING: NvmeScsi: 172: SCSI opcode 0x16 (0x45ba7628c5c8) on path vmhba65:C0:T2:L23966 to namespace eui.###################### failed with NVMe error status: 0x1 translating to SCSI e
2024-08-12T16:38:47.732Z cpu39:2100731 opID=16500354)WARNING: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0
2024-08-12T16:38:47.732Z cpu0:2097337)ScsiDeviceIO: 4167: Cmd(0x45ba7628c5c8) 0x16, CmdSN 0x2d277 from world 0 to dev "eui.######################" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0
2024-08-12T16:38:47.734Z cpu39:2100731 opID=16500354)LVM: 11050: LVMProbeDevice failed on (3811234624, eui.######################:1): Device does not contain a logical volume
2024-08-12T16:38:47.734Z cpu39:2100731 opID=16500354)FSS: 2472: Failed to create FS on dev [eui.######################:1] fs [SAMPLEDATASTORE] type [vmfs6] fbSize 1048576 => Not supported
2024-08-12T16:41:55.665Z cpu11:2098759)NvmeDiscover: 671: controller = nqn.1988-11.com.####:####:00:################vmhba64#58ccf090c9600f67:58ccf09849680f67
Environment
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x
Cause
In case of NVMe we only support ATS locking scheme as there is no equivalent to SCSI-2 PR and VMFS does not use SCSI-3 PR/NVMe reservations for managing devices.
In this case , HardwareAcceleratedLocking is disabled and hence VMFS switches to SCSI-2 that is not supported by NVMe targets and fails to mount.
Resolution
- Check if the ATS Locking is set to Enable or Disabled.
esxcli system settings advanced list --option /VMFS3/HardwareAcceleratedLocking
[root@ABC-VMW-S600:~] esxcli system settings advanced list --option /VMFS3/HardwareAcceleratedLocking
Path: /VMFS3/HardwareAcceleratedLocking
Type: integer
Int Value: 0 >> 0 is disabled and 1 is enable
Default Int Value: 1
Min Value: 0
Max Value: 1
String Value:
Default String Value:
Valid Characters:
Description: Enable hardware accelerated VMFS locking (requires compliant hardware).
- Check if the devices can support ATS or SCSi by running the command "esxcli storage vmfs lockmode list"
Reference: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.storage.doc/GUID-267970EC-38BA-47D1-8253-FF8A2C60FE3B.html - Run the command to enable ATS lock mechanism after device state is confirmed.
esxcli system settings advanced set --int-value 1 --option /VMFS3/HardwareAcceleratedLocking
[root@ABC-VMW-S600:~] esxcli system settings advanced list --option /VMFS3/HardwareAcceleratedLocking
Path: /VMFS3/HardwareAcceleratedLocking
Type: integer
Int Value: 1 >> 0 is disabled and 1 is enable
Default Int Value: 1
Min Value: 0
Max Value: 1
String Value:
Default String Value:
Valid Characters:
Description: Enable hardware accelerated VMFS locking (requires compliant hardware).
Post changing the ATS lock mechanism you should be able to mount the datastore presented via NVMe controller.
Additional Information
Disabling hardware accelerated locking (ATS) in ESXi
https://knowledge.broadcom.com/external/article/319797/disabling-hardware-accelerated-locking-a.html
Feedback
