VM guest OS I/O stall during iSCSI path failover

Products

• VMware vSphere ESXi

Issue/Introduction

• A VM guest OS experiences I/O freeze during iSCSI path failover.
• Other paths to the device remain available.

Environment

VMware vSphere ESXi 8.x

Cause

During iSCSI path failover device queue depth may become exhausted with pending I/O on the failing path. No more commands will be issued to the device until the iSCSI session recovery timeout period and path failover happens.

With default iSCSI adapter parameters iSCSI path failover can take from 25 to 35 seconds. I/O may freeze if queue depth becomes full with pending I/O on the failed path during this period.

To list the iSCSI adapter parameters:

#esxcli iscsi adapter param get -A vmhba##

Name                 Current     Default     Min     Max        Settable  Inherit
---------------      -------     -------     ----   ----        --------  -------
NoopOutInterval       15          15          1       60            true    false
NoopOutTimeout        10          10          10      30            true    false
RecoveryTimeout       10          10          1      120            true    false

 

Resolution

Reduce iscsi adapter param settings to minimum values from the ESXi command line.

esxcli iscsi adapter param set --adapter=<vmhba##> --key=NoopOutInterval --value=1  
esxcli iscsi adapter param set --adapter=<vmhba##> --key=NoopOutTimeout --value=10  
esxcli iscsi adapter param set --adapter=<vmhba##> --key=RecoveryTimeout --value=1

With these settings the maximum duration of I/O freeze will be 12 seconds.

Note from ESXi 9.1 the minimum values will be reduced further.