Storage keep ASYMMETRIC ACCESS STATE CHANGED will issued host lost storage connection

Products

• VMware vSphere ESXi

Issue/Introduction

Storage issued SCSI Valid sense data: 0x6 0x2a 0x6 (ASYMMETRIC ACCESS STATE CHANGED ) , meanwhile host lost the access to datastore . After a while ,  ASYMMETRIC ACCESS STATE CHANGED again , the datastore access been recovered .

 YYYY-MM-DDTHH:MM:SS cpu29:2098352)NMP: nmp_ThrottleLogForDevice:3875: H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x2a 0x6. Act:FAILOVER. cmdId.initiator=0x430991094740 CmdSN 0xfe6229e

..........

YYYY-MM-DDTHH:MM:SS cpu29:2098352)WARNING: NMP: nmp_DeviceStartLoop:740: NMP Device "naa.600a098038314653413f516758487855" is blocked. Not starting I/O from device.
YYYY-MM-DDTHH:MM:SS cpu29:2098352)WARNING: NMP: nmp_DeviceStartLoop:740: NMP Device "naa.600a098038314653413f516758487855" is blocked. Not starting I/O from device.

.........

YYYY-MM-DDTHH:MM:SS cpu46:2098869)NMP: nmp_ThrottleLogForDevice:3875: H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x2a 0x6. Act:FAILOVER. cmdId.initiator=0x430990ff2aa0 CmdSN 0x3f1c3b3
YYYY-MM-DDTHH:MM:SS cpu46:2098869)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.600a098038314463563f5167576d396b": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
 YYYY-MM-DDTHH:MM:SS cpu92:2098279)WARNING: NMP: nmpDeviceAttemptFailover:647: Retry world failover device "naa.600a098038314463563f5167576d396b" - issuing command 0x45d9d4d9b488
YYYY-MM-DDTHH:MM:SS cpu53:2098379)NMP: nmpCompleteRetryForPath:329: Retry world recovered device "naa.600a098038314463563f5167576d396b"

Environment

ESXi 7.x

ESXi 8.x

Cause

Storage side issued Valid sense data: 0x6 0x2a 0x6 (ASYMMETRIC ACCESS STATE CHANGED) , means it failover the Luns ,  ALUA provides a way of allowing devices report the state of its ports to hosts. This state can then be used by hosts to prioritize paths and make failover/load balancing decisions. 

Resolution

Need engage the storage vendor to check the storage device, what issued the failover and what negotiation information  sent to ESXi host .