Left Hand storage devices claimed by VMW_VAAIP_LHN fails to mount the ATS-ONLY VMFS datastores randomly.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
Random datastores fails to mount.

vmkernel traces : 

2019-12-17T19:15:16.774Z cpu60:66407)HPE_PSP_LH: psp_LHCommandComplete:2150: psp_LHSelectPath: cdb[0]:c1 serialNum:2
ESC[7m2019-12-17T19:15:16.774Z cpu60:66407)WARNING: HPE_SATP_LH: satp_lhn_pathFailure:1047: In satp_lhn_pathFailure status = 5 sense key = 20 and sense code = 0. path vmhba64:C3:T4:L0ESC[0m
ESC[7m2019-12-17T19:15:16.774Z cpu60:66407)WARNING: HPE_SATP_LH: satp_lhn_pathFailure:1048: path=vmhba64:C3:T4:L0 cmd[0]=c1 cmdid=2ESC[0m
ESC[7m2019-12-17T19:15:16.774Z cpu60:66407)WARNING: HPE_SATP_LH: satp_lhn_pathFailure:1194: In satp_lhn_pathFailure unknown failure.ESC[0m
2019-12-17T19:15:16.774Z cpu60:66407)ScsiDeviceIO: 2968: Cmd(0x439d413db640) 0xfe, CmdSN 0x2 from world 66139 to dev "naa.6000eb3f8ae176f9xxxxxxxx2344" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2019-12-17T19:15:16.774Z cpu64:66139)LVM: 6493: Forcing APD unregistration of devID 5bb3a695-42009326-e919-98f2b3020c5c in state 1.
2019-12-17T19:15:16.774Z cpu64:66139)LVM: 15081: Failed to open device naa.6000eb3f8ae176f9xxxxxxxx2344:1 : Not supported

VAAI status of the LUN : 

naa.6000eb3f8ae176f9xxxxxxxx2344
VAAI Plugin Name: VMW_VAAIP_LHN
ATS Status: supported
Clone Status: supported
Zero Status: supported
Delete Status: unsupported

Environment

VMware vSphere ESXi 6.5

Cause

This is a 3rd party issue. HPE-MEM plugIn is not working as expected for ATS commands for VMW_VAAIP_LHN plugin.

Resolution

To resolve this issue, the rules that allow the VMW_VAAIP_LHN plugin to claim these iSCSI sessions must be removed.

Step 1: On the ESXi host, verify the plugin list:

# esxcli storage core plugin registration list | grep -i LH
vmw_vaaip_lhn VMW_VAAIP_LHN VAAI
hpe-satp-lh HPE_SATP_LH SATP
hpe-psp-lh HPE_PSP_LH PSP

Step 2: Verify the VMW_VAAIP_LHN plugin has claimed the SAN volumes:

# esxcli storage core device vaai status get
naa.6000eb328ec1d2260000000000000015
VAAI Plugin Name: VMW_VAAIP_LHN
ATS Status: supported
Clone Status: supported
Zero Status: supported

Step 3: Remove the default VAAI and Filter claim rules:

# esxcli storage core claimrule remove -c VAAI --rule 65435

# esxcli storage core claimrule remove -c Filter --rule 65435

Step 4: Reboot the ESXi host.

Step 5: After reboot, verify the the VAAI plugin field is blank on the volume connections:

# esxcli storage core device vaai status get
naa.6000eb328ec1d2260000000000000015
VAAI Plugin Name: <blank> <== should be blank for your Lefthand devices
ATS Status: supported
Clone Status: supported
Zero Status: supported

This indicates the Native T10 plugin has ownserhip of the connections.

HPE advisory: https://support.hpe.com/hpesc/public/docDisplay?docId=a00101223en_us&docLocale=en_US

Workaround: