After applying the Cisco FPIN workaround we still see the dropped paths on the host

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

After applying the workaround in KB Temporary/transient storage path loss on ESXi 8.0 could result in paths not coming back when using Cisco UCS and NFNIC we are still dropping paths to storage. When we check the FPIN Heap, we find that we are still out of space after applying the workaround. You can check available FPINHeap with the following command:

esxcfg-info -a |grep -A3 storageFPINHeap|grep "Max Available"

Example:

Host-1 shows that it has run out of FPINheap. 

|----Max Available...................................416 bytes

Host-2 shows that we have not run out of Heap.

|----Max Available...................................3219872 bytes

 

 A healthy host will around 5246448 bytes Available but an impacted host will show signifyingly less free space sometimes 16k bytes or less. In this case, you have already disabled fpin but you still see that the Heap is empty (416 Bytes) 

Environment

• VMware ESXi 8.0U2
• VMware ESXi 8.0U3

Cause

There are 2 reasons that the heap is full despite disabling fpin: 

1. Before applying the workaround, we did not reboot. 

   • In this case, we never cleared the heap so Cisco cannot make a new path. Even with FPIN disabled, Cisco still needs some available heap to recreate a path. 
     
     
     

2. We are using ESXi 8.0 U2 and we rebooted after disabling fpin. 

   • The workaround for ESXi 8.0U2 is not persistent through a reboot so you need to disable fpin after every reboot
     • We suggest upgrading to ESXi 8.0U3 and disbling fpin again. 

Resolution

To resolve this situation, we will need to: 

1. Place the host into Maintenance Mode
2. Reboot the host
   1. If you are using ESXi 8.0U3 and have already run esxcli storage fpin info set -e false, then we only need to do a host reboot.  
   2. If you are using ESXi 8.0U2 you will need to reapply the workaround: 

      • • vsish -e set /storage/fpin/info 0

      • To check that this applied: 

        • esxcli storage fpin info get

3. Take the host out of Maintenance Mode 

Rebooting will clear the heap and this allows Cisco to create paths to storage again. 

Additional Information

For more information on this bug see: Temporary/transient storage path loss on ESXi 8.0 could result in paths not coming back when using Cisco UCS and NFNIC

There are two permanent fixes for this issue:

• A code fix to alter the FPIN open reference count behavior will be available in an upcoming ESXi 8.x release. (ESXi patch -- 8.0 P05 release)
• Cisco will be releasing NFNIC driver 5.0.0.46 that will change the driver behavior so it will use fixed Target IDs: https://bst.cisco.com/quickview/bug/CSCwn00553