On a switch reboot all iSCSI paths though multiple switches are lost

Products

• VMware vSphere ESXi

Issue/Introduction

• Rebooting a single network switch causes ESXi hosts to lose all iSCSI storage connections, despite having a redundant switch. This triggers an All Paths Down (APD) state and takes storage offline.
• iSCSI is configured as per best practices on the ESXi host and drivers/firmware are supported and up to date.

Environment

VMware vSphere ESXi 8.0

Cause

All paths are lost, and device encounters APD due to external physical network configuration responsible for iSCSI traffic. 

ESXi: /var/log/vmkernel.log 

####-##-##T##:##:##.###Z In(14) vobd[2097765]  [netCorrelator] 1033205274816us: [vob.net.dvport.uplink.transition.down] Uplink: vmnic2 is down. Affected dvPort: #/## ## ## ## ## # # #-## ## ## ## ## ## ## ##. 0 uplinks up. Failed criteria: 128
####-##-##T##:##:##.###Z In(14) vobd[2097765]  [netCorrelator] 1033205274874us: [vob.net.vmnic.linkstate.down]vmnic vmnic2 linkstate down

####-##-##T##:##:##.###Z In(14) vobd[2097765]  [scsiCorrelator] 1033236052653us: [vob.scsi.scsipath.pathstate.deadver2] scsiPath vmhba64:C2:T#:L# changed state from on (device ID: naa.################################)
####-##-##T##:##:##.###Z In(14) vobd[2097765]  [scsiCorrelator] 1033239103732us: [vob.scsi.scsipath.pathstate.deadver2] scsiPath vmhba64:C0:T#:L# changed state from on (device ID: naa.################################)
####-##-##T##:##:##.###Z In(14) vobd[2097765]  [APDCorrelator] 1033289112960us: [vob.storage.apd.start] Device or filesystem with identifier [naa.################################] has entered the All Paths Down state.
####-##-##T##:##:##.###Z In(14) vobd[2097765]  [scsiCorrelator] 1033289112876us: [vob.scsi.scsipath.pathstate.deadver2] scsiPath vmhba64:C1:T#:L# changed state from on (device ID: naa.################################)
####-##-##T##:##:##.###Z In(14) vobd[2097765]  [scsiCorrelator] 1033289112882us: [vob.scsi.scsipath.pathstate.deadver2] scsiPath vmhba64:C3:T#:L# changed state from on (device ID: naa.################################)

 

Resolution

Contact your network vendor to investigate unexpected APD, review switch configuration, cable length and traffic routing to the secondary switch that stayed active during the event.