ESXi hosts become unresponsive due to devices not entering APD/PDL state

Products

• VMware vSphere ESXi

Issue/Introduction

• Due to planned maintenance, some devices presnted to ESXi host become inaccessible
  
  
• Subsequently, ESXi hosts become unresponsive - e.g. VMs on accessible datastores cannot be managed via the UI.
  
  
• On restarting management agents via DCUI, the hosts do not become responsive/connected. A hard-reboot is required.
  
  
• Some ESXi hosts become disconnected in vCenter, but may recover without intervention.

Environment

VMware vSphere ESXi (all versions)

Cause

• This arises if the devices do not enter an All Paths Down (APD) or Permanent Device Loss (PDL) state because, when I/O fails on the devices, the Test Unit Ready (TUR) issued to test the path status succeeds. (i.e. the device does not process I/O but is not fully inaccessible).
  
  
• This leads to a repeated pattern of failed I/O, followed by a successful probe of the path, further failed I/O, etc.
  
  
• Over time, ongoing I/O failures cause degradation of the ESXi hostd service. 

Resolution

Engage your storage vendor to determine why the devices are not fully inaccessible.