VMs facing latency on Guest OS level due to fabric layer issues
search cancel

VMs facing latency on Guest OS level due to fabric layer issues

book

Article ID: 392802

calendar_today

Updated On: 06-04-2025

Products

VMware vSphere ESXi

Issue/Introduction

Symptom:

  • Latency seen on the Application layer from Guest OS.

  • Datastore size for all volumes presented from SAN box is 0.

 

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

 

Cause

  • Host to storage connectivity is disrupted and the issue is caused on the fabric layer as detected from FPIN congestion logging.

    In the '/var/run/log/vmkernel.log', we see below events -

    2025-04-18T15:24:29.355Z Wa(180) vmkwarning: cpu14:2097840)WARNING: VMW_SATP_ALUA: satp_alua_getTargetPortInfo:190: Could not get page 83 INQUIRY data for path "vmhba3:C0:T2:L2" - Timeout (195887137)
    2025-04-18T15:24:31.228Z Wa(180) vmkwarning: cpu52:2098424)WARNING: lpfc : vmhba3 lpfc_els_rcv_fpin_cgn:7266: 4657 FPIN CONGESTION WARNING Notification type Credit Stall (x2) Event Duration 10000 mSecs
    2025-04-18T15:24:31.228Z In(182) vmkernel: cpu52:2098424)StorageFPIN: 1279: Report FC FPIN Congestion Credit Stall event (hostWWPN 100000109b9cea58 tgtWWPN 2001d039eaa8b929) to vobd. 4 events have occurred since last report.
    2025-04-18T15:24:31.228Z In(182) vmkernel: cpu52:2098424)StoragePath: 5394: Calling MPP NMP for link event 2 on adapter vmhba3 (hostWWPN=0x100000109b9cea58 targetWWPN=0xffffffffffffffff targetNum = 4294967295)
    2025-04-18T15:24:31.228Z Wa(180) vmkwarning: cpu52:2098424)WARNING: NMP: nmpHandleLinkEvent:3998: Marking path vmhba3:C0:T3:L19 flaky on link event 2 with timeoutMS = 20000 flakyMarkTC = 81858609902804, reEvalFlakyPathTime = 20000
    2025-04-18T16:35:49.696Z Wa(180) vmkwarning: cpu112:2099458)WARNING: iodm: vmk_IodmEvent:191: vmhba4: FRAME DROP event has been observed 6 times in the last one minute. This suggests a problem with Fibre Channel link/switch!.

  • FPIN (Fabric Performance Impact Notifications) capability was added in ESXi 8.0 U2 to be able to better understand fabric related issues/events. This module will also print to /var/log/vmkernel.log when there are fabric events happening. The events that FPIN tracks and will report on are:
      • Link Integrity
      • Delivery
      • Congestion
      • Peer Congestion

Resolution