VMs become hung and datastores become inaccessible due to scsi reservation conflicts

Products

• VMware vSphere ESX 8.x
• VMware vSphere ESXi

Issue/Introduction

• Multiple production VMs are down.
• Hosts reporting as not responding in VC.
• Datastores are reporting as 0 in capacity and 0 in Free space

Hosts and Clusters - Cluster - Datastores tab

Environment

VMware vSphere ESXi

Cause

These symptoms arise if I/O is failing on VMFS datastores due to SCSI reservation conflicts triggered when ATS hardware accelerated locking is unexpectedly disabled on an ESXi host. 

ATS HardwareAcceleratedLocking is the standard and expected locking mechanism in use on ESXi hosts.

SCSI reservation-based locking is less efficient than ATS HardwareAcceleratedLocking and SCSI reservation conflicts may arise under higher I/O loads

The log message below is an example of the log message you will see when a scsi reservation conflict is reported. 

ESXi: /var/run/log/vmkernel.log
#-##-##T##:##:##.###Z cpu##:########)NMP: nmp_ResetDeviceLogThrottling:3778: Error status H:0x0 D:0x18 P:0x0 Sense Data: 0x0 0x0 0x0 from dev "naa.################################" occurred 2906 times(of 2906 commands)

To identify which devices are currently reporting scsi reservation conflicts:
dmesg | grep -i "D:0x18" | awk {'print $16'} | sort | uniq -c
    871 "naa.################################"
    870 "naa.################################"

 

To check if ATS locking is enabled globally on your ESXi host, run:
localcli system settings advanced list -o /VMFS3/HardwareAcceleratedLocking

SettingsAdvancedOption:
   Path: /VMFS3/HardwareAcceleratedLocking
   Type: integer
   Int Value: 1
   Default Int Value: 1
   Min Value: 0
   Max Value: 1
   String Value:
   Default String Value:
   Valid Characters:
   Description: Enable hardware accelerated VMFS locking (requires compliant hardware). Please see http://kb.vmware.com/kb/2094604 before disabling this option.

Resolution

To enable ATS locking globally on your ESXi host (this is the default setting):
esxcli system settings advanced set --int-value 1 --option /VMFS3/HardwareAcceleratedLocking

If all ESXi hosts that have access to this device have ATS locking enabled and the environment is continuing to encounter scsi reservation conflicts,
Then check storage presentation to ensure no non ESXi initiators have access to the storage device.

 

 

Additional Information

To check the locking mechanism that a VMFS datastore uses, use this command:
esxcli storage vmfs lockmode list

See: Display VMFS Locking Information