ESXi host may crash with PSOD with the message: BlueScreen: NOT_IMPLEMENTED bora/vmkernel/main/world.c:2307

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• PSOD with the message: BlueScreen: NOT_IMPLEMENTED bora/vmkernel/main/world.c:2307
• PSOD with the message: @BlueScreen: NMI IPI: Panic requested by another PCPU. RIPOFF(base):RBP:CS
• ALERT in the vmkernel.log similar to:

    ALERT: Heap: 2746: Unable to complete wait for non-empty heap     (worldGroup.2611820): Timeout

• PSOD backtrace for this issue will look like the following:

Backtrace for current CPU #8, worldID=2097296, fp=0x0
0x45390391bc00:[0x420038114d31]PanicvPanicInt@vmkernel#nover+0x1f5 stack: 0x100, 0x420038114d31, 0x0, 0x420000000001, 0x420038114d31
0x45390391bcb0:[0x4200381153a0]Panic_NoSave@vmkernel#nover+0x4d stack: 0x45390391bd10, 0x45390391bcd0, 0x420038427e15, 0x42003879d696, 0x8f6
0x45390391bd10:[0x4200381158ad]Panic_OnAssertAt@vmkernel#nover+0xba stack: 0x8f600000000, 0x42003879d696, 0x42003878ba99, 0x42003879d287, 0x4200387a8f5e
0x45390391bd90:[0x42003816855f]Int6_UD2Assert@vmkernel#nover+0x260 stack: 0x0, 0x0, 0x0, 0x420038161068, 0x0
0x45390391bdc0:[0x420038161067]gate_entry@vmkernel#nover+0x68 stack: 0x0, 0xbad0021, 0x420041000208, 0xbad0003, 0x1
0x45390391be80:[0x420038147136]World_DestroyHeap@vmkernel#nover+0x4e stack: 0x43167da00000, 0x420038147252, 0x4, 0x4200387dcb20, 0x45390391bef0
0x45390391bea0:[0x420038147251]WorldGroupCleanup@vmkernel#nover+0xe6 stack: 0x45390391bef0, 0x4200380f1def, 0x4301aa401234, 0x4301a8001220, 0x431149601220
0x45390391bec0:[0x4200380f1dee]InitTable_Cleanup@vmkernel#nover+0x27 stack: 0x431149601220, 0x42003814cd47, 0xffffffffffffffff, 0x4301aa201678, 0x45391579f000
0x45390391bee0:[0x42003814cd46]World_TryReap@vmkernel#nover+0x3d3 stack: 0x45391579f000, 0x45390391bf00, 0x0, 0x0, 0x0
0x45390391bfa0:[0x420038117582]ReaperWorkerWorld@vmkernel#nover+0xaf stack: 0x45390389f100, 0x45390391f100, 0x0, 0x0, 0x0
0x45390391bfe0:[0x420038428eca]CpuSched_StartWorld@vmkernel#nover+0x7b stack: 0x0, 0x4200380d788c, 0x0, 0x0, 0x0
0x45390391c000:[0x4200380d788b]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0, 0x0, 0x0, 0x0, 0x0
VMware ESXi 8.0.0 [Releasebuild-20513097 x86_64] NOT_IMPLEMENTED bora/vmkernel/main/world.c:2307

Note: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi 8.0.x
VMware vSphere ESXi 7.0.x

Cause

In a rare condition, vmkpollcontext object leak occurs in the vmkernel which eventually results in a PSOD.

iLO interrupt storm.

Resolution

Any HPE Gen10 / Gen10 Plus / Gen11 platforms running VMware ESXi 7.0 and ESXi 8.0 (or later) May Experience a PSOD With a "BlueScreen: NOT_IMPLEMENTED bora/vmkernel/main/world.c:2307" Message.

 

This issue is fixed by updating the HPE iLO Native driver component to v10.8.2 (or later).

 

For HPE platforms running VMware ESXi 7.0 (or later)

Download and install the following HPE iLO Native driver component:

HPE iLO Native Driver for ESXi 7.0 Version 10.8.2 (or later)

 

For HPE platforms running VMware ESXi 8.0 (or later)

Download and install the HPE iLO Native driver component v10.8.2 (or later), and the VMware ESXi 8.0 U2 kernel patch 80u2b, both available at the following URLs:

HPE iLO Native Driver for ESXi 7.0 Version 10.8.2 (or later)

Ref Advisory link from HPE for this PSOD : https://support.hpe.com/hpesc/public/docDisplay?docId=a00138303en_us

Additional Information

Impact/Risks:
No performance impact is associated with the workaround.
Hardware related events can be monitored through ILO or external monitoring software if any, vCenter/ESXi will not get any hardware health related information.

If the workaround is being applied, additional applications will need to be removed (HPE AMS/iSUT/iLORest). The removal of these applications will have the following impact due to compensatory effects.

• OS upgrade/SW/FW updates via VMware vLCM-HSM method will be impacted.
• HPE SPP/SSP online update method will be impacted.
• Losing information from ESXi in the iLO, this impact's HPE system management.
• Customer may lose automation ability for jobs/scripts which needs the information from AMS (SNMP) and iLORest.

Note: These impacts need to be evaluated before any action is taken.