Datastore Latency Events reported on Veeam server
search cancel

Datastore Latency Events reported on Veeam server

book

Article ID: 393820

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

  • Latency detected on storage.

  • Virtual Machines performance degrades.

  • Third-party monitoring tools (e.g., Veeam) will generate a high number of tickets related to datastore latency.

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

Datastore latency observed due to a high number of frame drops detected on the VMHBA.

Cause validation:

  • Verify the paths configured for the affected datastore using the command : "esxcfg-mpath -b -d naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx".

    esxcfg-mpath -b -d naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx : DGC Fibre Channel Disk (naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx)
       vmhba3:C0:T1:L11 LUN:11 state:active fc Adapter: WWNN: xx:xx:xx:xx:xx:xx:xx:xx WWPN: xx:xx:xx:xx:xx:xx:xx:xx  Target: WWNN: xx:xx:xx:xx:xx:xx:xx:xx WWPN: xx:xx:xx:xx:xx:xx:xx:xx
       vmhba3:C0:T0:L11 LUN:11 state:active fc Adapter: WWNN: xx:xx:xx:xx:xx:xx:xx:xx WWPN: xx:xx:xx:xx:xx:xx:xx:xx  Target: WWNN: xx:xx:xx:xx:xx:xx:xx:xx WWPN: xx:xx:xx:xx:xx:xx:xx:xx
       vmhba4:C0:T1:L11 LUN:11 state:active fc Adapter: WWNN: xx:xx:xx:xx:xx:xx:xx:xx WWPN: xx:xx:xx:xx:xx:xx:xx:xx  Target: WWNN: xx:xx:xx:xx:xx:xx:xx:xx WWPN: xx:xx:xx:xx:xx:xx:xx:xx
       vmhba4:C0:T0:L11 LUN:11 state:active fc Adapter: WWNN: xx:xx:xx:xx:xx:xx:xx:xx WWPN: xx:xx:xx:xx:xx:xx:xx:xx  Target: WWNN: xx:xx:xx:xx:xx:xx:xx:xx WWPN: xx:xx:xx:xx:xx:xx:xx:xx

    In the above example, the datastore is configured with 4 paths: 2 paths use vmhba3 and the other 4 use vmhba4.

  • The /var/run/log/vmkernel.log file indicates frame drops for vmhba3.
    YYYY-MM-DDTHH:MM.SSSZ cpu30:2097182)qlnativefc: vmhba3(63:0.0): qlnativefcStatusEntry:2071:scsi(0:44) Dropped frame(s) detected (255 of 255 bytes) residual length mismatch.
    YYYY-MM-DDTHH:MM.SSSZ cpu55:2097207)qlnativefc: vmhba3(63:0.0): qlnativefcStatusEntry:2071:scsi(1:44) Dropped frame(s) detected (255 of 255 bytes) residual length mismatch.
    YYYY-MM-DDTHH:MM.SSSZ cpu13:2098199)qlnativefc: vmhba3(63:0.0): qlnativefcStatusEntry:2071:scsi(1:8) Dropped frame(s) detected (255 of 255 bytes) residual length mismatch.
    YYYY-MM-DDTHH:MM.SSSZ cpu57:2097209)qlnativefc: vmhba3(63:0.0): qlnativefcStatusEntry:2105:(1:18) Dropped frame(s) detected (16384 of 16384 bytes).
    YYYY-MM-DDTHH:MM.SSSZ cpu57:2097209)qlnativefc: vmhba3(63:0.0): qlnativefcStatusEntry:2105:(1:18) Dropped frame(s) detected (131072 of 131072 bytes).
    YYYY-MM-DDTHH:MM.SSSZ cpu15:4704977)qlnativefc: vmhba3(63:0.0): qlnativefcStatusEntry:2071:scsi(1:18) Dropped frame(s) detected (255 of 255 bytes) residual length mismatch.
    YYYY-MM-DDTHH:MM.SSSZ cpu57:2097209)qlnativefc: vmhba3(63:0.0): qlnativefcStatusEntry:2105:(1:18) Dropped frame(s) detected (131072 of 131072 bytes).
    YYYY-MM-DDTHH:MM.SSSZ cpu57:2097209)qlnativefc: vmhba3(63:0.0): qlnativefcStatusEntry:2105:(1:18) Dropped frame(s) detected (129024 of 131072 bytes).
  • Verify the frame drops for the affected VMHBA using the command : "esxcli storage san fc events get".

    esxcli storage san fc events get
    FC Event Log
    ------------
    2025-04-07 08:58:11.912 [vmhba3] Dropped frames (255 of 115 bytes) on C0:T1:L0 cmd:0x12
    2025-04-07 08:58:12.112 [vmhba3] Dropped frames (129024 of 115 bytes) on C0:T1:L0 cmd:0x28
    2025-04-07 08:58:12.112 [vmhba3] Dropped frames (255 of 115 bytes) on C0:T1:L0 cmd:0x12
    2025-04-07 08:58:12.412 [vmhba3] Dropped frames (131072 of 115 bytes) on C0:T1:L0 cmd:0x28
    2025-04-07 08:58:12.712 [vmhba3] Dropped frames (131072 of 115 bytes) on C0:T1:L0 cmd:0x28
    2025-04-07 08:58:13.012 [vmhba3] Dropped frames (129024 of 115 bytes) on C0:T1:L0 cmd:0x28
    2025-04-07 08:58:13.312 [vmhba3] Dropped frames (122880 of 115 bytes) on C0:T1:L0 cmd:0x28
    2025-04-07 08:58:13.612 [vmhba3] Dropped frames (124928 of 115 bytes) on C0:T1:L0 cmd:0x28

Resolution

Involve switch and hardware vendor to validate the health of the affected driver and the ports.

Additional Information

Validate the driver and firmware compatibility. Please refer Compatibility Guide

Powered by Wolken Software