VMs experiencing latency at the Guest OS level due to Fabric Layer issues
search cancel

VMs experiencing latency at the Guest OS level due to Fabric Layer issues

book

Article ID: 392802

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

ESXi hosts may experience periodic or temporary loss of access to shared VMFS datastores, leading to performance degradation or unresponsiveness in virtual machines. This behavior is often observed in environments utilizing Fibre Channel (FC) storage where underlying fabric congestion occurs.

Symptoms:

  • High latency or "stuttering" observed within the Application layer of the Guest Operating System
  • Storage devices may appear as having a capacity of "0" for volumes presented from the SAN.
  • Virtual machines may become unresponsive or experience significant I/O wait times.
  • Loss of access to VMFS volumes with the error: Lost access to volume <UUID> due to connectivity issues.

Environment

  • VMware ESXi 8.x

Cause

  • The issue is caused by physical or fabric-layer congestion/disruption between the ESXi host and the storage array. Specifically, FPIN (Fabric Performance Impact Notifications) detects congestion such as "Credit Stalls" or "Frame Drops."

  • FPIN (Fabric Performance Impact Notifications) capability was added in ESXi 8.0 U2 to be able to better understand fabric related issues/events. This module will also print to /var/log/vmkernel.log when there are fabric events happening. The events that FPIN tracks and will report on are:
    • Link Integrity
    • Delivery
    • Congestion
    • Peer Congestion

  • In the /var/run/log/vmkernel.log file, the following warning patterns are logged
    vmkwarning: cpu##:#######)WARNING: VMW_SATP_ALUA: satp_alua_getTargetPortInfo:190: Could not get page 83 INQUIRY data for path "vmhba#:C#:T#:L#" - Timeout (##########)
    vmkwarning: cpu##:#######)WARNING: lpfc: vmhba# lpfc_els_rcv_fpin_cgn:7266: 4657 FPIN CONGESTION WARNING Notification type Credit Stall (x#) Event Duration ##### mSecs
    vmkernel: cpu##:#######)StorageFPIN: 1279: Report FC FPIN Congestion Credit Stall event (hostWWPN ################ tgtWWPN ################) to vobd. # events have occurred since last report.
    vmkernel: cpu##:#######)StoragePath: 5394: Calling MPP NMP for link event # on adapter vmhba# (hostWWPN=################## targetWWPN=################## targetNum = ##########)
    vmkwarning: cpu##:#######)WARNING: NMP: nmpHandleLinkEvent:3998: Marking path vmhba#:C#:T#:L## flaky on link event # with timeoutMS = ##### flakyMarkTC = ################, reEvalFlakyPathTime = #####
    vmkwarning: cpu###:#######)WARNING: iodm: vmk_IodmEvent:191: vmhba#: FRAME DROP event has been observed # times in the last one minute. This suggests a problem with Fibre Channel link/switch!.

Resolution

This is a physical infrastructure or fabric-level issue and needs vendor engagement to perform the following steps:

  1. Verify Fabric Health: Coordinate with your Storage/Fabric administrator to inspect the Fibre Channel switches for port errors, CRC errors, or SFP failure.
  2. Check SFP/Cabling: Inspect and potentially replace the SFPs or fiber cables connected to the HBA (vmhbaX) identified in the logs.
  3. Firmware/Drivers: Ensure that the HBA (Host Bus Adapter) is running on the recommended firmware and driver version listed in the Compatibility Guide.

Additional Information

FPIN (Fabric Performance Impact Notifications) was significantly enhanced in ESXi 8.0 U2. This module allows the ESXi kernel to receive notifications directly from the fabric switches when congestion occurs, allowing the host to make more informed decisions regarding path selection (NMP/PSP).

For more information on troubleshooting storage performance, see:

Powered by Wolken Software