FC LUN Path status does not get updated

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• When performing FC SAN maintenance or while having an unexpected storage path outage, once the storage path is back up on the ESXi host the paths still show as dead.

• Newly presented FC LUNs are not visible.

• Unable to unmount/delete VMFS datastore which are backed by FC LUN.

• Storage rescan of host fails with error: "An error occurred while communicating with the remote host."

Validation:

• The following log entry related to StorageFPIN  is seen multiple times in /var/log/vmkernel.log:

  WARNING: StorageFPIN: 521: Failed to allocate memory.

• The available FPINHeap on host can be checked with the following command.  A healthy host will around 5246448 bytes available, but an impacted host will show significantly less free space; sometimes 16k bytes or less:
  
  esxcfg-info -a |grep -A3 storageFPINHeap|grep "Max Available"
  
  For example, on on the following host 1 we have hit the issue and run out of FPINheap:
  |----Max Available...................................416 bytes
  
  On host 2 we have not yet run out of Heap:
  |----Max Available...................................3219872 bytes

Environment

VMware ESXi 8.0U2

Cause

FPIN (Fabric Performance Impact Notifications) capability was added to ESXi 8.0 U2 to be able to better understand fabric related issues. Due to a bug in the StorageFPIN code, when FPIN tries to allocate memory and is unable to, it can hold onto a reference count on the paths which prevents the FC vmhba driver from being able to allocate new paths or re-establish existing ones.

Resolution

This issue will be fixed in ESXi 8.0 P05 patch release. 

Workaround:

• To work around this issue, it is recommended to disable FPIN on ESXi 8.0 hosts which have FC storage attached.

• For ESXi 8.0.2:

  • Run the command in SSH to disable FPIN: vsish -e set /storage/fpin/info 0

  • To confirm the setting: vsish -e get /storage/fpin/info

  • This command is not persistent across reboots.

• For ESXi 8.0.3:

  • Run the command in SSH to disable FPIN: esxcli storage fpin info set -e false

  • To confirm the setting: esxcli storage fpin info get

 

Note: This setting change does not require a reboot, however if an ESXi host is already in a memory heap exhaustion state for storageFPINHeap then rebooting the host is required after making this setting change.