Ongoing APD (all paths down) issues across fiber channel impacting the ESXi host connectivity to storage

Products

• VMware vSphere ESXi

Issue/Introduction

Checking fiber channel events shows this occurring across only a single HBA, example below shows dropped frames on only vmhba3

 

# localcli storage san fc events get

 

[root@<hostname>:/var/log] localcli storage san fc events get
FC Event Log
------------
YYYY-MM-DD HH:12:09.303 [vmhba3] Dropped frames (925696 of 774 bytes) on <target/lun id> cmd:0x28 
YYYY-MM-DD HH:12:09.607 [vmhba3] Dropped frames (645120 of 774 bytes) on <target/lun id> cmd: 0x28 
YYYY-MM-DD HH:12:09.607 [vmhba3] Dropped frames (514048 of 774 bytes) on <target/lun id> cmd:0x28 
YYYY-MM-DD HH:12:09.607 [vmhba3] Dropped frames (948224 of 774 bytes) on <target/lun id> cmd:0x28 
YYYY-MM-DD HH:12:09.911 [vmhba3] Dropped frames (1044480 of 774 bytes) on <target/lun id> cmd: 0x28 
YYYY-MM-DD HH:12:09.911 [vmhba3] Dropped frames (780288 of 774 bytes) on <target/lun id> cmd: 0x28 
YYYY-MM-DD HH:12:09.911 [vmhba3] Dropped frames (989184 of 774 bytes) on <target/lun id> cmd:0x28 
YYYY-MM-DD HH:12:10.214 [vmhba3] Dropped frames (907264 of 774 bytes) on <target/lun id> cmd:0x28 
YYYY-MM-DD HH:12:10.214 [vmhba3] Dropped frames (800768 of 774 bytes) on <target/lun id> cmd:0x28 
YYYY-MM-DD HH:12:10.214 [vmhba3] Dropped frames (454656 of 774 bytes) on <target/lun id> cmd:0x28 
YYYY-MM-DD HH:12:10.518 [vmhba3] Dropped frames (923648 of 774 bytes) on <target/lun id> cmd:0x28

 

 

The same dropped frames are also visible from vmkernel.log and only across vmhba3 when counting all instances of dropped frames per HBA

 

# grep Dropped /var/log/vmkernel.log |awk '{print $3}' |sort |uniq -c

 

YYYY-MM-DDTHH:17:08.561z cpu23:2097502) qlnativefc: vmhba3 (##:0.0): qlnativefcStatusEntry:1922: (8:104) Dropped frame (s) detected (1015808 of 1048576 bytes). 
YYYY-MM-DDTHH:17:08.5612 cpu23:2097502) qlnativefc: vmhba3 (##:0.0): qlnativefcStatusEntry:1922: (1:4) Dropped frame (s) detected (1046528 of 1048576 bytes). 
YYYY-MM-DDTHH:17:08.5612 cpu23:2097502) qlnativefc: vmhba3 (##:0.0): qlnativefcStatusEntry:1922: (8:105) Dropped frame (s) detected (804864 of 1048576 bytes). 
[root@<hostname>:/var/log] grep Dropped vmkernel.log | awk '{print $3}' sort uniq -c 
        352 vmhba3 (##:0.0):

Environment

7.0

Resolution

Disable HBA paths from out-of-band management interface (iDRAC, iLO, BMC, etc).  If this option is not available, then disable all paths to the impacted HBA vi CLI

 

Check all HBA path states

# localcli storage core path list

 

Disable all paths for a specific HBA, example below for vmhba3

# localcli storage core path list |grep "Runtime Name:" |grep vmhba3 |awk '{print $3}' |while read line; do localcli storage core path set --path $line --state off; done

 

Confirm the paths show "State: off" or "State: dead" and reboot the host