Lost access to VMFS datastores on IBM FlashSystem during rolling upgrades

Products

• VMware vCenter Server

Issue/Introduction

ESXi hosts report "Lost access to volume" alerts for multiple VMFS datastores during a rolling microcode upgrade or node reboot of an IBM FlashSystem (e.g., FS9500) 

The following patterns appear in the host logs:

• vmkernel.log: Rapid succession of fabric Registered State Change Notifications (RSCNs) followed immediately by device loss timers. ####-##-##T##:##:##.###Z In(182) vmkernel: cpu##:#######)lpfc: lpfc_els_rcv_rscn:####: vmhba# #### RSCN received Data: ####
####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu##:#######)WARNING: lpfc : vmhba# lpfc_start_devloss:####: #### Start 10 sec devloss tmo WWPN ##:##:##:##:##:##:##:##

• Native Multipathing (NMP): Paths become blocked as the driver queue saturates during fabric discovery loops. ####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu#:#######)WARNING: NMP: nmp_DeviceStartLoop:###: NMP Device "naa.################################" is blocked. Not starting I/O from device.

• SCSI Errors: Heartbeat operations fail with MISCOMPARE status due to I/O timeouts. ####-##-##T##:##:##.###Z In(182) vmkernel: cpu##:#######)NMP: nmp_ThrottleLogForDevice:####: Cmd 0x89 (...) Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0. Act:NONE. SK=MISCOMPARE

Environment

ESXi 8.0 U3
IBM FlashSystem storage
Emulex HBAs

Cause

Rolling storage node reboots trigger multiple fabric RSCNs. The default ESXi configuration lacks the necessary thresholds to maintain connectivity during these transitions:

1. Driver Timeout: The default Emulex lpfc_devloss_tmo of 10 seconds is too short. The host preemptively removes paths while the fabric is still settling from the reboot.
2. Path Selection Policy: The default Round Robin (RR) policy locks I/O to a single path for 1,000 operations. If that path stalls due to fabric discovery, I/O does not immediately fail over to a surviving node, causing metadata heartbeats (VAAI ATS) to time out.

Resolution

Optimize the ESXi host multipathing and driver parameters to increase patience during storage maintenance.

Step 1: Increase Emulex Devloss Timeout

Adjust the Emulex driver threshold to 30 seconds to prevent premature path eviction during fabric discovery storms.

1. Run the following command on each host: esxcli system module parameters set -m lpfc -p "lpfc_devloss_tmo=30"

   ⚠️ IMPORTANT: This operation requires a scheduled host reboot to take effect.

Step 2: Optimize Round Robin IOPS Limit

Modify the path selection policy to rotate paths after every single I/O operation. This ensures immediate utilization of surviving paths if a controller node becomes unavailable.

1. Add a new claim rule for the IBM storage array: esxcli storage nmp satp rule add -s VMW_SATP_ALUA -v IBM -m 2145 -P VMW_PSP_RR -o "iops=1" -c "user_default" -d "Optimized for IBM FlashSystem HyperSwap"

   ⚠️ Review this command with your storage administrator before running it to ensure compatibility with your specific array model.