Error: 'SCSI Reservation Conflict', Datastore Fails to Mount with Clustered VMDK Feature Enabled.
search cancel

Error: 'SCSI Reservation Conflict', Datastore Fails to Mount with Clustered VMDK Feature Enabled.

book

Article ID: 387812

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

  • Once the Clustered VMDK feature is enabled, the datastore becomes disconnected and fails to mount, displaying the following error:
    "SCSI reservation conflict."

  • ESXi hosts enter a "Not responding" state.

Steps to Validate:

  • Validate Whether the Clustered VMDK feature is enabled: 

Click on vCenter >> (Inventory) Datastore >> Datastore_name >> Configure >> General >> Datastore Capabilities >> Clustered VMDK (Enabled)


  • Validate if the Host is getting into  "Not Responding" state:

 

  • Review the /var/run/log/vmkernel.log from the ESXi host. The following entries will be identified, indicating a failure to reclaim resources due to a SCSI reservation conflict:

YYYY-02-07T17:43:25.110Z cpu35:2100411)FDS: 1120: Persistent reserve failed on naa.################################:1; status = SCSI reservation conflict
YYYY-02-07T17:44:05.129Z cpu23:2097247)ScsiDeviceIO: 4154: Cmd(0x45d953878f88) 0x89, cmdId.initiator=0x4308589ab140 CmdSN 0x4 from world 2100411 to dev "naa.################################" failed H:0x5 D:0x0 P:0x0 . Cmd count Active:0 Queued:0
YYYY-02-07T17:44:12.301Z cpu7:2097890)NMP: nmp_ResetDeviceLogThrottling:3778: Error status H:0x0 D:0x18 P:0x0 Sense Data: 0x0 0x0 0x0 from dev "naa.################################" occurred 2246 times(of 2246 commands)
YYYY-02-07T17:44:45.139Z cpu24:2097250)ScsiDeviceIO: 4154: Cmd(0x45d953878a88) 0x5f, cmdId.initiator=0x4308589d1090 CmdSN 0x22c9 from world 0 to dev "naa.################################" failed H:0x5 D:0x0 P:0x0 . Cmd count Active:0 Queued:0

  • Review the /var/run/log/hostd.log. The following entries will be found, indicating the datastore could not be mounted, and the file system remains inaccessible.

YYYY-02-07T17:46:18.897Z info hostd[2100411] [Originator@6876 sub=Libs] UnresolvedVmfsVolume: deviceName=naa.################################:1,lvmName=########-fc808b3d-077d-#########,label=Datastore_name,fsUuid=######-5c436fca-b137-#########
YYYY-02-07T17:46:41.695Z info hostd[2101422] [Originator@6876 sub=Libs] UnresolvedVmfsVolume: deviceName=naa.################################:1,lvmName=########-fc808b3d-077d-#########,label=Datastore_name,fsUuid=######-5c436fca-b137-#########
YYYY-02-07T17:46:43.445Z info hostd[2101422] [Originator@6876 sub=Libs] GetUnmountedVmfsFileSystemsInt: uuid:######-5c436fca-b137-#########

 

Environment

VMware vSphere ESXi 7.x

Cause

  • As per the design, the Clustered VMDK feature should not be enabled on a production VMFS volume while virtual machines are in a "Powered On" state.
  • The perennially-reserved flag is set to false on datastores with the Clustered VMDK feature enabled for all ESXi hosts.

Note: This issue is not observed in ESXi version 8.0, where the 'perennially-reserved' flag is automatically set to 'True' upon activating Clustered VMDK Support.

Resolution

Sequence to Activate Clustered VMDK Support on an new VMFS6 datastore.

Step 1:

Refer below link to validate, if the LUN supports clustered VMDK
Clustered vmdk datastore LUN support check

Step 2:

Set  perennially reserved flag on datastores using below KB.
Device naa.id with a VMFS partition is marked perennially reserved in the vmkernel log

Run this command to set the flag:
esxcli storage core device setconfig -d naa.id --perennially-reserved=true

After enabling this flag you see below entries in vmkernel.log

Device naa.xxxxxxxxxxxx with a VMFS partition is marked perennially reserved. This is not supported and may lead to data loss

You can safely ignore this warning for Clustered VMDK datastores. VMware Engineering is working on suppressing the message in future release.
Reference KB: ESXi host takes a long time to start during rescan of RDM LUNs

Step 3:

Activate Clustered VMDK Support on an VMFS6 datastore.

Reference Doc: Activating Clustered VMDK Support

Step 4:

To search for new or establish existing VMFS datastores, run this command:
vmkfstools -V

 

Steps to fix SCSI reservation conflict post enabling Clustered VMDK Support where the perennially-reserved flag is set to "False".

This activity should be performed during a scheduled maintenance window.

Step 1:

Refer below link to validate, if the LUN supports clustered VMDK
Clustered vmdk datastore LUN support check

Step 2:

Select one host to remain powered on. Power off all other hosts in the cluster. If the datastore is attached to multiple clusters, power off all hosts in all those clusters. The goal is to have only one host with the LUN naa.############ attached powered on.

Step 3:

Set  perennially reserved flag on Clustered VMDK datastores using below KB.
Device naa.id with a VMFS partition is marked perennially reserved in the vmkernel log

Run this command to set the flag:
esxcli storage core device setconfig -d naa.id --perennially-reserved=true

After enabling this flag you see below entries in vmkernel.log

Device naa.xxxxxxxxxxxx with a VMFS partition is marked perennially reserved. This is not supported and may lead to data loss

You can safely ignore this warning for Clustered VMDK datastores. VMware Engineering is working on suppressing the message in future release.
Reference KB:  ESXi host takes a long time to start during rescan of RDM LUNs

Step 4:

To search for new or establish existing VMFS datastores, run this command:
vmkfstools -V

Note: If you're still facing an issue, please reach out to Broadcom Support if you're encountering problems with an existing setup where Clustered VMDK Support has already been activated.

Powered by Wolken Software