ESXi host takes more than 60 seconds to failover the path when storage process reboots on an EMC unity storage array

Products

• VMware vSphere ESXi

Issue/Introduction

During a Storage Process (SP) reboot on an EMC Unity storage array, ESXi host can take over 60 seconds to failover the path to the peer SP.

Cause

ESXi host probes and iterate through all the paths before finding the active path and doing a path failover. ESXi host internally uses mode sense and RTPG requests to probe the paths and each of these request uses 10 seconds of time out. In case of SP reboot, ESXi host does not get response while probing the path from the target and waits for the commands to time out. For each path the time out is around 20 seconds (10 seconds time out for mode sense and RTPG respectively).

 

The overall delay is directly proportional to the number of paths per SP. For example: If you have 4 paths per SP, then overall fail over time will be double of 2 paths per SP. Hitting a long time out is unlikely because the SP will force its link down, which triggers Registered State Change Notifications (RSCN) across the fabric and ESXi host fails over immediately.

 

Note: The impact is seen only if there are active IO in progress during SP reboot.

Resolution

This is a known issue affecting ESXi and EMC Unity storage array.

 

Currently, there is no resolution. However, EMC and VMware engineering teams are diligently working together on a solution.

 

To work around this issue, use these options:

• If an SP reboots, Vmware recommends to install VMware Tools on all the virtual machines housed on the EMC Unity array.
• Increase the disk timeout for the Windows virtual machine to 120.
• Edit the Registry key HKEY_LOCAL_MACHINE/System/CurrentControlSet/Services/Disk. For more information, see Inconsistent Windows virtual machine performance when disks are located on SAN datastores (1014).

Notes:

• This procedure modifies the Windows registry. Before making any registry modifications, ensure that you have a current and valid backup of the registry and the virtual machine. For more information on backing up and restoring the registry, see the Microsoft Knowledge Base article Windows registry information for advanced users.
• It is not required to increase the disk timeout for Linux VMs with VMware Tools as this has already been increased to 180. For more information, see Increasing the disk timeout values for a Linux 2.6 virtual machine (1009465).