FRAME DROP event has been observed

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Latency seen to storage.

• VMs are randomly impacted and VMs have outages or performance of the VMs is degraded. 

vmkernel.log reporting:

 [YYYY-MM-DDTHH:MM:SS] cpu35:2115960)WARNING: iodm: vmk_IodmEvent:194: vmhba1: FRAME DROP event has been observed 400 times in the last one minute. This suggests a problem with Fibre Channel link/switch!.
 [YYYY-MM-DDTHH:MM:SS] cpu34:2115856)WARNING: iodm: vmk_IodmEvent:194: vmhba1: FRAME DROP event has been observed 400 times in the last one minute. This suggests a problem with Fibre Channel link/switch!.
 [YYYY-MM-DDTHH:MM:SS] cpu32:2145125)WARNING: iodm: vmk_IodmEvent:194: vmhba2: FRAME DROP event has been observed 400 times in the last one minute. This suggests a problem with Fibre Channel link/switch!.

 [YYYY-MM-DDTHH:MM:SS] cpu2:2098292)ScsiDeviceIO: 4124: Cmd(0x45b981710948) 0x28, CmdSN 0x22a2 from world 2100337 to dev "naa.624a93######f4b840001" failed H:0x7 D:0x28 P:0x0
 [YYYY-MM-DDTHH:MM:SS] cpu6:2098268)ScsiDeviceIO: 4124: Cmd(0x45b9816c0e48) 0x28, CmdSN 0x22ac from world 2100337 to dev "naa.624a93######e00011ad2" failed H:0x7 D:0x28 P:0x0
 [YYYY-MM-DDTHH:MM:SS] cpu2:2098292)ScsiDeviceIO: 4124: Cmd(0x45b9c47ee788) 0x28, CmdSN 0x22a9 from world 2100337 to dev "naa.624a93######400011ad2" failed H:0x7 D:0x28 P:0x0

[YYYY-MM-DDTHH:MM:SS] cpu10:2098284)WARNING: ScsiDeviceIO: 1513: Device naa.624a93######f4b840001 performance has deteriorated. I/O latency increased from average value of 914 microseconds to 18292 microseconds.
[YYYY-MM-DDTHH:MM:SS] cpu32:2098290)WARNING: ScsiDeviceIO: 1513: Device naa.624a93######f4b840001 performance has deteriorated. I/O latency increased from average value of 914 microseconds to 36950 microseconds.
[YYYY-MM-DDTHH:MM:SS] cpu0:2098288)WARNING: ScsiDeviceIO: 1513: Device naa.624a93######f4b840001 performance has deteriorated. I/O latency increased from average value of 914 microseconds to 18407 microseconds.

In Qlogic FC-HBA: 

 [YYYY-MM-DDTHH:MM:SS] In (182) vmkernel: cpu54:2098257) qlnativefc: vmhba2 (12:0.0): qlnativefcStatusEntry:1919: (7:41) Dropped frame (s) detected (106496 of 131072 bytes). 
 [YYYY-MM-DDTHH:MM:SS] In (182) vmkernel: cpu54:2098257) qlnativefc: vmhba2 (12:0.0): qlnativefcStatusEntry: 2067:C0:T7:L41 FCP command status: 0x15-0x0 (0x2) portid=bc0142 oxid=0x4ba c 80000 len-131072 rspInfo=0x0   resid=0x0 fwResid=0x1a000 host status = 0x2 device  


  [YYYY-MM-DDTHH:MM:SS] In (182) vmkernel: cpu54:2098257) qlnativefc: vmhba2 (12:0.0): qlnativefcStatusEntry:1919: (7:41) Dropped frame (s) detected (116736 of 131072 bytes).
 [YYYY-MM-DDTHH:MM:SS] In (182) vmkernel: cpu54:2098257) qlnativefc: vmhba2 (12:0.0): qlnativefcStatusEntry:2067:C0:T7:L41 FCP command status: 0x15-0x0 (0x2) portid=bc0142 oxid=0x4bc c 80000 len-131072 rspInfo=0x0 resid=0x0  fwResid=0x1c800 host status = 0x2 device st$


• You may also see the "Dropped frames" when running the command below:

esxcli storage san fc events get

 

Example of output:

 

[YYYY-MM-DDTHH:MM:SS] [vmhbaX] Dropped Frames (262144 of 165 bytes) on C0:T0:L0 cmd:0x8a

[YYYY-MM-DDTHH:MM:SS] [vmhbaX] Dropped Frames (131072 of 165 bytes) on C0:T0:L0 cmd:0x8a

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

As message states this indicates some problem on the physical fiber channel link ( Fiber-optic cable) or the SFP on FC-Switch connected to this host.

FRAME DROP events suggests a possible connectivity issue or congestion. An erroneous FC Port can also lead to this issue.

Host Status [H:0x7] :- This status is returned when a device has been reset due to a Storage Initiator Error. This typically occurs due to an outdated HBA firmware or possibly (though rarely) as the result of a bad HBA.

For [H:0x7] refer article: Understanding SCSI host-side NMP errors/conditions in ESXi

Resolution

As this is outside ESXi scope, engage with the fabric/hardware/storage vendor to investigate and mitigate the dropped frames in the environment.