iSCSI Storage Multipathing and Boot LUN Availability during Cisco UCS Fabric Interconnect Upgrades

Products

• VMware vSphere ESXi

Issue/Introduction

This article addresses concerns regarding the storage stability of VMware ESXi hosts during a rolling upgrade of Cisco UCS Fabric Interconnects (FI). In environments where hosts are configured to boot from iSCSI LUNs with paths striped across redundant fabrics (Fabric A and Fabric B), there is often uncertainty regarding whether a host can survive the loss of 50% of its storage paths without a reboot.

Observed Symptoms during Maintenance (if not optimized):

• Log messages in /var/log/vmkernel.log indicating: ScsiDeviceIO: 4033: Command 0x2a to device "naa.xxx" failed H:0x1 D:0x0 P:0x0 Possible Loss of Path.
• Increased I/O latency (DAVG) during the transition phase.
• Potential for Guest OS file systems to transition to "Read-Only" if failover timers are not properly aligned.
• ention specifically that Guest OS file systems may transition to "Read-Only" if failover timers are not aligne

Environment

Hypervisor: VMware ESXi 6.7, 7.x, or 8.x.

Hardware: Cisco UCS B-Series or C-Series Servers.

Storage Configuration: Boot-from-iSCSI LUNs.

Network Topology: Redundant iSCSI paths across isolated VLANs, each pinned to a specific Fabric Interconnect.

Cause

The potential for host instability during fabric maintenance is not caused by the loss of the physical path itself, but by a mismatch between hardware recovery times and software storage timeouts.

Path Selection Delay: By default, VMware may use a "Fixed" pathing policy. If the "Preferred Path" resides on the fabric being rebooted, the host must manually transition to a standby path. If this transition is not instantaneous, I/O timeouts occur.

Aggressive iSCSI Timers: The default RecoveryTimeout for iSCSI initiators may be shorter than the time required for a Cisco UCS Fabric Interconnect to complete its reset and re-initialize the logical links.

Multipathing Logic: VMware Native Multipathing (NMP) relies on the Pluggable Storage Architecture (PSA). If the host is not configured to "Round Robin" between all available paths, it may not efficiently utilize the surviving fabric's bandwidth, leading to I/O congestion during the upgrade window.

 

Resolution

To ensure non-disruptive operations, the environment must be configured to utilize Round Robin pathing with optimized I/O limits and extended session timers. These settings should be applied via VMware Host Profiles to ensure cluster-wide consistency.

1. Modify Host Profile Advanced Options

Edit the Host Profile associated with the cluster and navigate to the following paths:

• iSCSI Login Timeout:

  • Path:  Storage configuration > iSCSI Software Adapter > iSCSI adapter > Advanced Options

  • Value: Set LoginTimeout to 30 seconds. This allows sufficient time for ARP resolution and session re-establishment after the fabric returns.

• iSCSI Recovery Timeout:

  • Path: Same as above.

  • Value: Set RecoveryTimeout to 60 seconds. This keeps the I/O queue "alive" while the fabric transitions, preventing the host from declaring a Permanent Device Loss (PDL).

2. Configure Round Robin and IOPS Limit

Ensure that the host checks path health and distributes I/O every single command:

• Path Selection Policy: Set the default policy for the storage array's SATP (Storage Array Type Plugin) to VMW_PSP_RR.

• IOPS Limit: Under Device Specific Selection Policy, ensure the IOPS limit is set to 1. This ensures that if a path on Fabric A fails, the very next I/O command is instantly redirected to Fabric B.

3. UCS Fabric Evacuation (Pre-Maintenance)

Prior to initiating the firmware reboot in UCS Manager (UCSM), utilize the Fabric Evacuation feature:

1. Navigate to Equipment > Fabric Interconnects > Fabric Interconnect A.

2. Select Evacuate Fabric.

3. Monitor the ESXi hosts. This gracefully drains traffic to the alternate fabric, allowing you to verify host stability before the FI physically reboots.

4. Verify MTU consistency (e.g., 9000/8972 for Jumbo Frames) across the ESXi vSwitch, UCS QoS, and Storage Array

4. Verification Script (PowerCLI)

Execute the following script to ensure all hosts have correctly inherited the settings:

PowerShell

Get-Cluster "YourCluster" | Get-VMHost | ForEach-Object {
    $esxcli = Get-EsxCli -VMHost $_ -V2
    $esxcli.storage.nmp.device.list.Invoke() | Select-Object @{N="Host";E={$_.VMHost.Name}}, Device, PathSelectionPolicy
}

Effectiveness: These steps transition the storage failover from a "reactive" hardware event to a "proactive" software-managed process. By extending timeouts and forcing frequent path switching (IOPS=1), the ESXi host remains unaware of the underlying maintenance, maintaining continuous I/O flow to the iSCSI LUNs.

To identify your Storage Array Type Plugin (SATP), you need to see which specific driver VMware has assigned to your LUNs. This is critical because Host Profile rules for "Round Robin" and "IOPS=1" are often applied based on the SATP name (e.g., VMW_SATP_ALUA or VMW_SATP_DEFAULT_AA).

Identifying your SATP

Run the following command on any host in the cluster:

esxcli storage nmp device list

Look for the line Storage Array Type. Common values include:

• VMW_SATP_ALUA: Used for most modern mid-to-high-end arrays (NetApp, EMC, Pure).

• VMW_SATP_DEFAULT_AA: Used for Active-Active arrays.

• VMW_SATP_DEFAULT_AP: Used for Active-Passive arrays.

Applying the Rule to the Host Profile

Once you have the SATP name, follow these steps to ensure every LUN from that vendor uses the optimized settings:

1. Open Host Profile: Go to Storage configuration > Native Security Path Selection Policy.

2. Add Path Selection Option: * SATP: Enter the name you found above (e.g., VMW_SATP_ALUA).

   • PSP: Set to VMW_PSP_RR.

3. Add Device Specific Rule:

   • Under the Round Robin Action section, ensure the IOPS value is set to 1.

Summary Checklist

Action Item	Method	Targeted Value
LoginTimeout	Host Profile (Advanced)	30
RecoveryTimeout	Host Profile (Advanced)	60
Path Policy	Host Profile (SATP Rule)	VMW_PSP_RR
Switching frequency	Host Profile (PSP Rule)	IOPS = 1
Fabric Maintenance	UCS Manager	Fabric Evacuation: Enabled

Final Verification before Upgrade

After applying the Host Profile and remediating the hosts, perform a manual rescan. If the hosts are correctly configured, you should be able to run this command and see the number 1 for the "iops" limit on every LUN:

esxcli storage nmp psp roundrobin deviceconfig get -d <naa.ID>