"All shared datastores failed" alarm on reboot of physical FC switches with APD

Products

• VMware vSphere ESXi

Issue/Introduction

• In a vSphere environment connected to Fibre Channel storage, physical switches are rebooted one at a time (e.g. as part of an upgrade sequence).
  
  
• The first switch reboot completes.
  
  
• However, when the second switch goes down, access is lost to the datatsores on ESXi hosts, with vCenter alarm:
  
  
  • "All shared datastores failed"

Environment

VMware vSphere ESXi (all versions)

Cause

• Shortly after the first switch has rebooted and paths via the switch come up, the ESXi hosts receive a process logout request from the fabric/targets. Paths via the switch are lost.
  
  
• When the second switch is taken down, ESXi hosts lose the remaining paths and devices enter an All Paths Down (APD) state.
  
  
  
• Sample logging (will vary with driver):
  
  

First switch goes down and come back up:
####-##-##T##:##:##.###Z In(182) vmkernel: cpu17:2098078)nfnic: <2>: INFO: fnic_handle_link: 1075: fnic2: recvd up to down event
...
####-##-##T##:##:##.###Z In(182) vmkernel: cpu17:2098078)nfnic: <2>: INFO: fnic_fdls_link_status_change: 98: fnic2: FDLS link status change link up:1, usefip:0

A fabric login (FLOGI) is issued followed by a process login (PRLI):
####-##-##T##:##:##.###Z In(182) vmkernel: cpu17:2098078)nfnic: <2>: INFO: fdls_send_fabric_flogi: 982: Sending fabric FLOGI for wwpn:0x20000025b513b05c Setting FLOGI MFS to 2048
####-##-##T##:##:##.###Z In(182) vmkernel: cpu5:2098079)nfnic: <2>: INFO: fdls_process_tgt_prli_rsp: 1839: PRLI accepted from target: 0x1a0120
####-##-##T##:##:##.###Z In(182) vmkernel: cpu5:2098079)nfnic: <2>: INFO: fdls_process_tgt_prli_rsp: 1902: PRLI: Target found: 0x1a0120. TGT now in ready state. Adding tport. 

Log out requests are received from targets, eg.:
####-##-##T##:##:##.###Z In(182) vmkernel: cpu5:2098079)nfnic: <2>: INFO: fdls_process_logo_req: 3640: Process LOGO request from fcid: 0xb20000
####-##-##T##:##:##.###Z In(182) vmkernel: cpu5:2098079)nfnic: <2>: INFO: fnic_tport_exch_reset: 4883: Tport exch reset: target id: 39 tport->fcid: 0xb20000
####-##-##T##:##:##.###Z In(182) vmkernel: cpu5:2098079)nfnic: <2>: INFO: fnic_fdls_remove_tport: 414: Removed tport from deleted list tid:39

Second switch goes down:
####-##-##T##:##:##.###Z In(182) vmkernel: cpu69:2098071)nfnic: <1>: INFO: fnic_handle_link: 1001: link status 0 down cnt 2
####-##-##T##:##:##.###Z In(182) vmkernel: cpu69:2098071)nfnic: <1>: INFO: fnic_handle_link: 1003: old status 1 old down cnt 1
####-##-##T##:##:##.###Z In(182) vmkernel: cpu69:2098071)nfnic: <1>: INFO: fnic_handle_link: 1075: fnic1: recvd up to down event

APD is triggered, e.g.:
####-##-##T##:##:##.###Z In(182) vmkernel: cpu51:2097748)StorageApdHandler: 1191: APD start for 0x43096abf29d0 [naa.#################################]
####-##-##T##:##:##.###Z In(182) vmkernel: cpu29:2097742)StorageApdHandler: 408: APD start event for 0x43096abf29d0 [naa.#################################]
####-##-##T##:##:##.###Z In(182) vmkernel: cpu29:2097742)StorageApdHandlerEv: 106: Device or filesystem with identifier [naa.#################################] has entered the All Paths Down state.

Resolution

Contact the fabric and storage vendors for assistance.