FRAME DROP event has been observed
search cancel

FRAME DROP event has been observed

book

Article ID: 371539

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

  • Latency seen to storage.
  • VMs are randomly impacted.
  • vmkernel.log reporting:

 2024-07-25T18:00:36.102Z cpu35:2115960)WARNING: iodm: vmk_IodmEvent:194: vmhba1: FRAME DROP event has been observed 400 times in the last one minute. This suggests a problem with Fibre Channel link/switch!.
   2024-07-25T18:00:56.271Z cpu34:2115856)WARNING: iodm: vmk_IodmEvent:194: vmhba1: FRAME DROP event has been observed 400 times in the last one minute. This suggests a problem with Fibre Channel link/switch!.
   2024-07-25T18:00:56.862Z cpu32:2145125)WARNING: iodm: vmk_IodmEvent:194: vmhba2: FRAME DROP event has been observed 400 times in the last one minute. This suggests a problem with Fibre Channel link/switch!.

   2024-07-25T18:03:28.973Z cpu2:2098292)ScsiDeviceIO: 4124: Cmd(0x45b981710948) 0x28, CmdSN 0x22a2 from world 2100337 to dev "naa.624a93######f4b840001" failed H:0x7 D:0x28 P:0x0
   2024-07-25T18:03:28.973Z cpu6:2098268)ScsiDeviceIO: 4124: Cmd(0x45b9816c0e48) 0x28, CmdSN 0x22ac from world 2100337 to dev "naa.624a93######e00011ad2" failed H:0x7 D:0x28 P:0x0
   2024-07-25T18:03:28.973Z cpu2:2098292)ScsiDeviceIO: 4124: Cmd(0x45b9c47ee788) 0x28, CmdSN 0x22a9 from world 2100337 to dev "naa.624a93######400011ad2" failed H:0x7 D:0x28 P:0x0

 2024-07-25T11:01:03.333Z cpu10:2098284)WARNING: ScsiDeviceIO: 1513: Device naa.624a9370f0872ae3fbdf4b8400011ad4 performance has deteriorated. I/O latency increased from average value of 914 microseconds to 18292 microseconds.
   2024-07-25T11:01:19.886Z cpu32:2098290)WARNING: ScsiDeviceIO: 1513: Device naa.624a9370f0872ae3fbdf4b8400011ad4 performance has deteriorated. I/O latency increased from average value of 914 microseconds to 36950 microseconds.
   2024-07-25T11:02:02.393Z cpu0:2098288)WARNING: ScsiDeviceIO: 1513: Device naa.624a9370f0872ae3fbdf4b8400011ad4 performance has deteriorated. I/O latency increased from average value of 914 microseconds to 18407 microseconds.

  • You may also see the "Dropped frames" when running the command below:
esxcli storage san fc events get
 
Example of output:
 
2024-07-18 14:29:55.531 [vmhbaX] Dropped Frames (262144 of 165 bytes) on C0:T0:L0 cmd:0x8a
2024-07-18 14:30:57.074 [vmhbaX] Dropped Frames (131072 of 165 bytes) on C0:T0:L0 cmd:0x8a.
 
 

Environment

VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0

Cause

As message states this indicates some problem on the physical fibre channel link or on the switch connected to this host.

FRAME DROP events suggests a possible connectivity issue or congestion. An erroneous FC Port can also lead to this issue.

Host Status [H:0x7] :- This status is returned when a device has been reset due to a Storage Initiator Error. This typically occurs due to an outdated HBA firmware or possibly (though rarely) as the result of a bad HBA.

 For [H:0x7] ref - https://knowledge.broadcom.com/external/article/309543/understanding-scsi-hostside-nmp-errorsco.html

Resolution

As this is outside ESXi scope, engage with the fabric/hardware/storage vendor to investigate and mitigate the dropped frames in the environment.
 
 
 
 
 
Powered by Wolken Software