Storage adapter rescan and LUN discovery fail due to Storage FPIN heap exhaustion on ESXi 8.0 U3d and earlier
search cancel

Storage adapter rescan and LUN discovery fail due to Storage FPIN heap exhaustion on ESXi 8.0 U3d and earlier

book

Article ID: 380321

calendar_today

Updated On:

Products

VMware vSphere ESXi 8.0

Issue/Introduction

Symptoms:

  • After performing a unmount and detach of the datastore, rescanning HBA task fails with error 'an error occurred while communicating with the remote host'.

 

  • Validation of /var/log/vmkernel.log confirms that the Cisco Fibre Channel adapter detects a link failure and immediately performs a firmware reset.

2026-05-21T08:08:28.428Z INFO: fnic_handle_link: link status 0 down cnt 2
2026-05-21T08:08:28.428Z INFO: fnic_handle_link: fnic2: recvd up to down event
2026-05-21T08:08:28.428Z INFO: fnic_fdls_link_down: Moving tport to PLOGO state tport:5
2026-05-21T08:08:28.428Z INFO: fnic_fdls_link_down: Moving tport to PLOGO state tport:8
2026-05-21T08:08:28.428Z WARNING: fnic_fw_reset_handler: Issued fw reset
2026-05-21T08:08:28.430Z INFO: fnic_fcpio_cmpl_handler: received reset cmpl handler

  • Validation of /var/log/vmkernel.log confirms that Native Multipathing (NMP) blocks I/O while attempting storage path failover.

2026-05-21T08:08:28.430Z NMP: Cmd failed on path vmhba2:C0:T15:L2
2026-05-21T08:08:28.430Z WARNING: Device "naa.###########": awaiting fast path state update for failover with I/O blocked.
2026-05-21T08:08:29.270Z WARNING: Retry world failover device "naa.###########"
2026-05-21T08:08:29.270Z NMP: Retry world recovered device "naa.###########"

  • Validation of /var/log/vmkernel.log confirms that storage path scan and claim operations return BUSY after the Fibre Channel interruption.

2026-05-21T08:12:39.049Z WARNING: vmk_ScsiScanAndClaimPaths returned BUSY
2026-05-21T08:12:40.049Z WARNING: vmk_ScsiScanAndClaimPaths returned BUSY
2026-05-21T08:12:44.050Z WARNING: lunmap update failed,retry ..

  • Validation of /var/log/vmkernel.log confirms that LUN discovery fails because storage paths cannot be reclaimed.

2026-05-21T08:12:56.052Z WARNING: lun add failure! in_remove:0 ioAllowed:1
2026-05-21T08:12:56.052Z WARNING: lunmap update failed,retry ..

  • Validation of /var/log/vobd.log confirms that multiple Fibre Channel paths become unavailable, resulting in degraded path redundancy.

2026-05-21T08:08:28.438Z Path redundancy to storage device naa.########### degraded.
2026-05-21T08:08:28.438Z Path vmhba2:C0:T13:L1 is down.

Environment

  • VMware ESXi 8.x

Cause

  • The storage interruption is caused by a known issue in the Storage Fabric Performance Impact Notification (FPIN) framework introduced in ESXi 8.0 U2 to monitor Fibre Channel fabric health.
  • During Fibre Channel path interruptions or SAN maintenance events, the Storage FPIN component attempts to allocate memory to process FPIN events. If memory allocation fails because the storageFPINHeap is exhausted, Storage FPIN retains stale references to the affected storage paths instead of releasing them.
  • The Cisco nfnic driver creates new storage path objects each time Fibre Channel connectivity is re-established after a path loss. Since the stale path references are still held by Storage FPIN, the driver cannot successfully create or reclaim new storage paths. As additional path reconnect events occur, the stale references continue to accumulate, eventually preventing new paths from being established.
    • As a result:
      • Storage path scanning returns BUSY.
      • LUN discovery and path claiming fail.
      • Native Multipathing (NMP) performs repeated storage failover attempts.
      • Storage devices may temporarily enter an All Paths Down (APD) state until the host is rebooted or the issue is otherwise mitigated.
  • This behavior is a known issue affecting ESXi 8.0 U3d and lower version systems using the Storage FPIN feature together with the Cisco nfnic Fibre Channel driver.
  • Validation of /var/log/vmkernel.log confirms that StorageFPIN cannot allocate memory, preventing FPIN processing.

2026-05-21T08:22:20.354Z WARNING: StorageFPIN: Failed to allocate memory.
2026-05-21T08:22:21.141Z WARNING: vmk_ScsiScanAndClaimPaths returned BUSY

  • Validation of /var/log/vmkernel.log further confirms that the StorageFPIN heap reaches its maximum size and cannot expand further.

2026-05-21T15:22:22.908Z WARNING: Heap storageFPINHeap already at its maximum size. Cannot expand.
2026-05-21T15:22:22.908Z WARNING: StorageFPIN: Failed to allocate memory.

  • Validate the FPIN runtime statistics using vsish -e get /storage/fpin/fc/stats. The command returns Out of memory, confirming that the storageFPINHeap is exhausted and Storage FPIN is unable to allocate additional memory to process Fibre Channel path events. This directly supports the known issue where stale FPIN references prevent storage paths from being reclaimed or re-established.

# vsish -e get /storage/fpin/fc/stats

VSISHCmdGetInt():Get failed: Out of memory
Error: Error in command cat: Out of memory

Resolution

There is currently only one fix for this issue:

  • A code fix to alter the FPIN open reference count behavior is now available in ESXi 8.0 U3e (build 24674464)
  • If an upgrade cannot be performed immediately, reboot the affected ESXi host to clear the exhausted storageFPINHeap and release stale storage path references.
Powered by Wolken Software