LUNs attached to VMware vSphere 6.0 hosts may remain in APD Timeout state after paths have recovered

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• When an APD event occurs, LUNs connected to ESXi may remain inaccessible after paths to the LUNs recover.
• The 140-second APD timeout expires even when paths to storage are recovered.
• In the /var/log/vmkernel.log file, you experience these events in sequence:
   
  1. Device enters APD.
  2. Device exits APD.
  3. Heartbeat recovery and filesystem operations on the device fail due to timeout or not found or busy.
  4. The APD timeout expires despite the fact that the device exited APD previously.
      
• This condition is associated with one or more of these behaviors:
   
  • Virtual machines becomes inaccessible.
  • Hosts becomes unresponsive.
  • Storage is not online, even though paths are up and available.
  • Datastore disappears from the vSphere Client, even when virtual machines on that datastore remain.
     
• An APD event can be triggered by one or more of these events. This list is not exhaustive:
   
  • Failures of upstream Fibre Channel or Ethernet switches in such a way that affect all paths to the storage array
  • Storage array failure or reboot
  • Storage array firmware updates (some vendors)

Important: Not all APD events exhibit this behavior. In most cases, LUNs and datastores exit the APD condition normally and as expected.

Environment

VMware vSphere ESXi 6.0

Cause

This issue occurs due to a fault in APD handling. When this issue occurs, LUN paths are available and online during an APD event, but the APD timer continues upcounting until the LUN enters APD Timeout state. After the initial APD event, the datastore is inaccessible as long as active workloads are associated with the datastore.

Resolution

This issue is resolved in ESXi 6.0 Update 1, available at VMware Downloads. For more information, see the VMware ESXi 6.0 Update 1 Release Notes.

 

If you are unable to upgrade, there are no workarounds that can guarantee that this issue is not encountered during an APD event. However, there are two workarounds to restore production should this issue occur.

 

To work around the issue, use one of these options:

• Perform the procedure to stop all outstanding I/O to the LUN. For information on unplanned PDL, see Cannot remount a datastore after an unplanned permanent device loss (PDL) (2014155).
  
  Note: It may also be necessary to restart ESXi management agents. For more more information, see Restarting the Management agents on an ESXi or ESX host (1003490).
   
• Reboot all hosts with volumes in the APD Timeout state.

Additional Information

For more information regarding APD events, see:

• Storage device has entered the All Paths Down state (2032934)
• Storage device has recovered from the APD state (2032945)
• All Paths Down timeout for a storage device has expired (2032940)

Storage device has entered the All Paths Down state
All Paths Down timeout for a storage device has expired
Storage device has recovered from the APD state
连接到 VMware vSphere 6.0 主机的 LUN 在恢复路径后可能仍保持 APD 超时状态
VMware vSphere 6.0 ホストに接続されている LUN がパスのリカバリ後も APD タイムアウト状態のままとなる

Impact/Risks:

• When this issue is encountered, virtual machines must be terminated to recover the datastore.
   
  • HA, if enabled, should recover these virtual machines on other hosts.
     
• If management agents must be restarted, the host temporarily lose manageability through vCenter Server.