Storage Center systems with Front End SAS connectivity show lun capacity 0MB

Products

VMware vSphere ESXi

Issue/Introduction

In the Dell EMC Storage Compatibility Matrix, the 6.5 VMWare ESXi version is tested / supported with SAS front end connectivity. SAS driver "lsi-msgpt3" Version 12 and SCOS Versions 7.2.x and 7.3.x were included in this tested and supported solution. However, upgrades to VMWare ESXi 6.5 U1, 6.5 U2, and 6.7 automatically install updated lsi-msgpt3 SAS drivers, such as Version 16. Without manual intervention of configuring a module parameter on the driver, lsi-msgpt3 versions greater than 12 do not work properly (e.g., rebooting host sees LUN capacity of 0 bytes, lose access to SAS Front End volumes upon controller failover, PSOD kernel dumps) with ESXi 6.5 and SCOS.

Symptoms:

[root@vmserver3:~] esxcfg-scsidevs -c
Device UID                            Device Type      Console Device                                            Size      Multipath PluginDisplay Name
naa.6000d310041df2000000000000000005  Direct-Access    /vmfs/devices/disks/naa.6000d310041df2000000000000000005  0MB       NMP     VMFS5
naa.6000d310041df2000000000000000007  Direct-Access    /vmfs/devices/disks/naa.6000d310041df2000000000000000007  0MB       NMP     VMFS02
naa.6000d310041df2000000000000000009  Direct-Access    /vmfs/devices/disks/naa.6000d310041df2000000000000000009  0MB       NMP     VMFS41
naa.6000d310041df200000000000000000a  Direct-Access    /vmfs/devices/disks/naa.6000d310041df200000000000000000a  1048576MB NMP     VMFS6
naa.6000d310041df200000000000000000b  Direct-Access    /vmfs/devices/disks/naa.6000d310041df200000000000000000b  0MB       NMP     VMFS7

### Devices are visible but size is shown as 0MB

sas.5d0946606ea42c00-sas.5000d310041df21e-naa.6000d310041df200000000000000000b:
   Runtime Name: vmhba4:C0:T1:L7
   Device: naa.6000d310041df200000000000000000b
   Device Display Name: VMFS7
   Group State: standby
   Array Priority: 0
   Storage Array Type Path Config: {TPG_id=61470,TPG_state=STBY,RTP_id=30,RTP_health=UP}
   Path Selection Policy Path Config: {current path; rank: 0}

### Standby Path is used for the current path

naa.6000d310041df200000000000000000b : VMFS7
   vmhba4:C0:T1:L7 LUN:7 state:standby sas Adapter: 5d0946606ea42c00  Target: 5000d310041df21e

naa.6000d310041df200000000000000000c : VMLOG01
   vmhba4:C0:T1:L8 LUN:8 state:active sas Adapter: 5d0946606ea42c00  Target: 5000d310041df21e

### In NMP path list host shows only one path for the device

2019-01-25T21:59:54.421Z cpu7:66233)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6000d310041df200000000000000000b": awaiting fast path state update before retrying failed command again...
2019-01-25T21:59:55.421Z cpu17:65820)WARNING: NMP: nmp_RegisterDeviceEvents:923: Could not register events 0x180 on device "naa.6000d310041df200000000000000000b", Status: Not found. Event Registration will be retried on next path eval.
2019-01-25T21:59:55.421Z cpu14:66417)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6000d310041df200000000000000000b" - issuing command 0x439d80fff3c0
2019-01-25T21:59:55.421Z cpu14:66417)WARNING: NMP: nmp_SelectPathAndIssueCommand:4633: PSP selected path "vmhba4:C0:T1:L7" in a bad state (standby) on device "naa.6000d310041df200000000000000000b".
2019-01-25T21:59:55.421Z cpu14:66417)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x12 (0x439d80fff3c0) to dev "naa.6000d310041df200000000000000000b" failed on path "vmhba4:C0:T1:L7" H:0x1 D:0x0 P:0x0 Invalid sense data: 0x0 0x0

### vmkernel logs shows clearly that NMP is trying to access device on standby path

Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7

Cause

Storage Center systems with Front End SAS connectivity have been officially tested and supported to work with ESXi version 6.7 .f you are encountering issues with ESXi 6.5 and SCOS 7.2 / 7.3, then the best practice setting is to modify the module parameter for all versions of the lsi-msgpt3 SAS driver. It is a requirement for any version greater than 12.

Resolution

List out the module parameter values

esxcli system module parameters list -m lsi_msgpt3

Modify the issue_scsi_cmd_to_bringup_drive module parameter

esxcli system module parameters set -p issue_scsi_cmd_to_bringup_drive=0 -m lsi_msgpt3

Validate that the module parameter value has been modified with the list command in Step 1
Reboot the ESXi host for the changed value to take effect.

Additional Information

https://www.dell.com/support/article/us/en/04/sln313031/sc-storage-customer-notification-driver-compatibility-with-front-end-sas-connectivity?lang=en

Impact/Risks:
Host is unable to mount the datastores after reboot