PSOD on ESXI 8.0 U3 for world running 3rd Party Multi Path Plugin
search cancel

PSOD on ESXI 8.0 U3 for world running 3rd Party Multi Path Plugin

book

Article ID: 387956

calendar_today

Updated On:

Products

VMware vSphere ESXi 8.0

Issue/Introduction

Symptoms:

  • With ESXi 8.0 U2, FPIN (Fabric Performance Impact Notifications) capability was added.
  • FPIN messages proactively alert devices within a Fabric network about specific conditions that may impact performance
  • With ESXi 8.0 U3, this capability was extended to the Multi Path Plugins (MPPs). The received notifications can now be forwarded to the MPPs
  • Partner plugins developed using ESXi SDK version lesser than 8.0U3 do not support FPIN handling.
  • In a in a SAN configuration where FPIN notifications are supported, ESXi host upon receiving FPIN notifications may crash with a PSOD if MPP developed using SDK version lesser than 8.0U3 is installed on the ESXi host.

Validation:

  • In /var/log/vmkernel.log, you would notice the below events when a FPIN is received:
[YYYY-MM-DDTHH:MM:SS ]cpu70:2098697)WARNING: lpfc: lpfc_els_rcv_fpin_cgn:7657: vmhba4 4657 FPIN CONGESTION WARNING Notification type Credit Stall (x2) Event Duration 10000 mSecs
[YYYY-MM-DDTHH:MM:SS] cpu70:2098697)StorageFPIN: 1279: Report FC FPIN Congestion Credit Stall event (hostWWPN #### tgtWWPN ####) to vobd. 0 events have occurred since last report.
[YYYY-MM-DDTHH:MM:SS] cpu70:2098697)StoragePath: 5394: Calling MPP PowerPath for link event 2 on adapter v 0x0

[YYYY-MM-DDTHH:MM:SS] cpu70:2098697)World: 3357: TR 0x#### GDT 0x#### (0x####) IDT 0x#### (0x####)
[YYYY-MM-DDTHH:MM:SS] cpu70:2098697)World: 3359: CR0 0x#### CR3 0x#### CR4 0x####
[YYYY-MM-DDTHH:MM:SS]  cpu109:2098704)WARNING: lpfc: lpfc_els_rcv_fpin_cgn:7657: vmhba6 4657 FPIN CONGESTION WARNING Notification type Credit Stall (x2) Event Duration 10000 mSecsESC[0m
[YYYY-MM-DDTHH:MM:SS] cpu109:2098704)StorageFPIN: 1279: Report FC FPIN Congestion Credit Stall event (hostWWPN #### tgtWWPN ####) to vobd. 0 events have occurred since last report.
[YYYY-MM-DDTHH:MM:SS] cpu109:2098704)StoragePath: 5394: Calling MPP PowerPath for link event 2 on adapter vmhba6 (hostWWPN=0x#### targetWWPN=0x#### targetNum = 4294967295)
[YYYY-MM-DDTHH:MM:SS] cpu109:2098704)World: 3355: PRDA 0x#### ss 0x0 ds 0x#### es 0x#### fs 0x#### gs 0x####
[YYYY-MM-DDTHH:MM:SS] cpu109:2098704)World: 3357: TR 0x#### GDT 0x#### (0x####) IDT 0x#### (0x####)
[YYYY-MM-DDTHH:MM:SS] cpu109:2098704)World: 3359: CR0 0x#### CR3 0x#### CR4 0x####YYYY-MM-DDTHH:MM:SS.SSSZ cpu70:2098697)WARNING: SelfDiagnostics: 550: Failed to do successful stack walk. Stack is corruptmhba4 (hostWWPN=0x#### targetWWPN=0x#### targetNum = 4294967295)
[YYYY-MM-DDTHH:MM:SS] cpu70:2098697)World: 3355: PRDA 0x#### ss 0x0 ds 0x#### es 0x#### fs 0x0 gs
  • Upon receiving the FPIN, the ESXi may crash for world running the MPP similar to the backtrace below:
[[YYYY-MM-DDTHH:MM:SS] cpu109:2098704)Backtrace for current CPU #109, worldID=2098704, fp=0x####
[YYYY-MM-DDTHH:MM:SS] cpu109:2098704)0x####:[0x####]PanicvPanicInt@vmkernel#nover+0x#### stack: 0x####, 0x####, 0x####, 0x####, 0x####
[YYYY-MM-DDTHH:MM:SS] cpu109:2098704)0x####:[0x####]Panic_ExceptionMsg@vmkernel#nover+0x#### stack: 0x####, 0x####, 0x####, 0x####, 0x####
[YYYY-MM-DDTHH:MM:SS] cpu109:2098704)0x####:[0x####]Panic_Exception@vmkernel#nover+0x#### stack: 0x####, 0x####, 0x####, 0x####, 0x####
[YYYY-MM-DDTHH:MM:SS] cpu109:2098704)0x####:[0x####]IDTReturnPrepare@vmkernel#nover+0x#### stack: 0x####, 0x####, 0x####, 0x####, 0x####
[YYYY-MM-DDTHH:MM:SS] cpu109:2098704)0x####:[0x####]gate_entry@vmkernel#nover+0x#### stack: 0x####, 0x####, 0x####, 0x####, 0x####
[YYYY-MM-DDTHH:MM:SS] cpu109:2098704)Panic: 783: Halting PCPU 109.
[YYYY-MM-DDTHH:MM:SS] cpu70:2098697)Backtrace for current CPU #70, worldID=2098697, fp=0x####
[YYYY-MM-DDTHH:MM:SS]ccpu70:2098697)VMware ESXi 8.0.3 [Releasebuild-24022510 x86_64]#PF Exception 14 in world 2098697:lpfc_do_work IP 0x0 addr 0x0

Environment

  • VMware vSphere ESXi 8.0 U3. 
  • SAN Fabric (FC switch) with FPIN notifications enabled.
  • Partner MPP compiled against VMKAPI SDK less than 8.0U3 OR Partner MPP that does not implement FPIN event handler.

Cause

  • MPPs developed using VMKAPI SDK version lesser than 8.0U3 do not have access to the new FPIN VMKAPIs definitions and it is also possible that MPPs compiled against latest SDK might not implement the event handler.
  • The ESXi host crashes with the PSOD, since ESXi 8.0 U3 and higher assumes that MPP always implements the event handler interface starting.

Resolution

This issue is addressed in vSphere ESXi 8.0 U3e Build 24022510.

Workaround:

  • Disable the FPIN notifications processing in the VMkernel storage layer by using the esxcli command line.
  • The command saves the FPIN activation to both ConfigStore and the VMkernel System Interface Shell and persists across ESXi reboots.
  • Following is the command used to activate or deactivate the Fabric Performance Impact Notification (FPIN).
    #esxcli storage fpin info set -e= <true/false>
  • Above configuration change can be verified using
    #esxcli storage fpin info get

Additional Information

ESXi 8.0 U3 でサードパーティ製マルチパスプラグインを実行しているワールドが原因で PSOD が発生する

Powered by Wolken Software