On failure of a storage controller, multiple devices go APD on multiple ESXi hosts

Products

• VMware vSphere ESXi

Issue/Introduction

• One storage controller on a storage array fails, or is otherwise taken down (e.g. for upgrade).
  
  
• In this scenario, half of the paths to devices are expected to become "Dead"
  
  
• However, multiple devices on multiple hosts enter an All Paths Down (APD) state.
  
  
• Prior to the failure there were no errors indicating failed paths ("Path redundancy degraded"). 

Environment

VMware vSphere ESXi (all versions)

Cause

• Certain devices only have paths available through the failed controller, due to device access configuration on the storage array.
  
  
• Loss of these paths cause the devices to enter an APD state.

Resolution

• Review device access configuration on the array (e.g. host initiator groups) and ensure the configuration is consistent for ESXi access to all devices.
  
  
• On Fibre Channel storage, ensure zoning has been correctly configured.