ESXi host experiences a PSOD with the message: Tried to unlock non-locked lock 0x432948b711d8

Products

• VMware vSphere ESXi

Issue/Introduction

ESXi Host crashes with a PSOD Error 'Tried to Unlock Non-Locked Lock' as shown below.

You may get similar Backtrace as shown below:

[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)Backtrace for current CPU #50, worldID=2097202, fp=0x0
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)0x4539c191b9a0:[0x42002cb7be80]PanicvPanicInt@vmkernel#nover+0x20c stack: 0x42004c806b48, 0x42002cb7be80, 0x0, 0x420000000001, 0x42002cb7be80
[YYYY-MM-DDTHH:MM:SS]cpu50:2097202)0x4539c191ba50:[0x42002cb7c5fc]Panic_NoSave@vmkernel#nover+0x4d stack: 0x4539c191bab0, 0x4539c191ba70, 0x46, 0x432948b711d8, 0x401
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)0x4539c191bab0:[0x42002ca39f96]vmk_SpinlockUnlock@vmkernel#nover+0x63 stack: 0x432948b71150, 0x42002db51e26, 0x0, 0x42002db590c6, 0x4539c191baf0
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)0x4539c191bad0:[0x42002db51e25]nmlx_Complete@(nmlx5_core)#<None>+0x1a stack: 0x4539c191baf0, 0x432948601480, 0x0, 0x0, 0x0
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)0x4539c191bae0:[0x42002db590c5]nmlx5_CompleteEnt@(nmlx5_core)#<None>+0x13e stack: 0x0, 0x0, 0x0, 0x1, 0x432948b71120
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)0x4539c191bb20:[0x42002db59a82]nmlx5_CmdCompHandler@(nmlx5_core)#<None>+0x127 stack: 0x4521c97da5c0, 0x432948601480, 0x42002dc01c40, 0x42002dbc554f, 0x80
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)0x4539c191bb60:[0x42002db5d195]nmlx5_MSIxISR@(nmlx5_core)#<None>+0x1fa stack: 0x4539c191bb88, 0x0, 0x4539c191bb90, 0x430286ccab90, 0x12e
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)0x4539c191bbc0:[0x42002cb5fa3b]IntrCookieBH@vmkernel#nover+0x170 stack: 0x430286cc3aa0, 0x1, 0x430286cc3aa0, 0x430286ccab20, 0x39
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)0x4539c191bc50:[0x42002cb3f98c]BH_DrainAndDisableInterrupts@vmkernel#nover+0x159 stack: 0x42004c801570, 0xe5890e297f9ee, 0x0, 0x100000000, 0x42004c801040
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)0x4539c191bcc0:[0x42002cb616ff]IntrCookie_VmkernelInterrupt@vmkernel#nover+0xa4 stack: 0x46, 0x42002d0a890b, 0x0, 0x0, 0x0
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)0x4539c191bce0:[0x42002d0a890a]IDT_IntrHandler@vmkernel#nover+0x97 stack: 0x0, 0x42002d0a10c7, 0x103, 0x0, 0x0
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)0x4539c191bd00:[0x42002d0a10c6]gate_entry@vmkernel#nover+0xa7 stack: 0x0, 0x0, 0x1757f, 0x414, 0x43028718bab0
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)0x4539c191bdc8:[0x42002ca90ea7]Power_ArchPerformWait@vmkernel#nover+0x157 stack: 0x42004c801880, 0x800000000, 0x100000414, 0x42004c800000, 0x42004c800000
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)0x4539c191bdd0:[0x42002ca90f75]Power_ArchSetCState@vmkernel#nover+0xba stack: 0x800000000, 0x100000414, 0x42004c800000, 0x42004c800000, 0x0
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)0x4539c191be20:[0x42002d0da1ed]CpuSchedIdleLoopInt@vmkernel#nover+0x292 stack: 0x0, 0x7fffffffffffffff, 0x1, 0x7fffffffffffffff, 0xfffffffffffffff6
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)0x4539c191be90:[0x42002d0dedc5]CpuSched_IdleLoop@vmkernel#nover+0x12 stack: 0x32, 0x42002ca804c8, 0x0, 0x0, 0x0
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)0x4539c191bea0:[0x42002cb5efaa]Init_APIdle@vmkernel#nover+0x3f stack: 0x0, 0x0, 0x0, 0x0, 0x0
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)0x4539c191beb0:[0x42002ca804c7]SMPAPIdle@vmkernel#nover+0x27c stack: 0x0, 0x0, 0x0, 0x0, 0x0
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)ESC[45mESC[33;1mVMware ESXi 8.0.3 [Releasebuild-24674464 x86_64]ESC[0m Tried to unlock non-locked lock 0x432948b711d8

 

Errors just before PSOD. (vmkernel.log)

[YYYY-MM-DDTHH:MM:SS] cpu58:2098008)WARNING: HBX: 2723: Failed to cleanup registration key on volume6849748c-efa69ae7-7246-5c2573e0e3c2: FailureESC[0m
[YYYY-MM-DDTHH:MM:SS] cpu58:2098008)WARNING: Vol3: 4678: Error closing the volume: . Eviction fails: FailureESC[0m
[YYYY-MM-DDTHH:MM:SS] cpu122:2100149)FSS: 7434: Failed to open file 'hpilo-d0ccb9'; Requested flags 0x5, world: 2100149 [sut], (Existing flags 0x5, world: 2101011 [ahsdv]): Busy
[YYYY-MM-DDTHH:MM:SS] cpu9:3622147)SchedVsi: 2208: Group: host/opt/vsan/vsanperfsvc(2756331): min=170 max=204, units: mb
[YYYY-MM-DDTHH:MM:SS] cpu74:2098732)Unmap6: 10536: datastore1: Acquired UC with 0 OP ucOffset 446464 ucIndex: 94
[YYYY-MM-DDTHH:MM:SS] cpu74:2098732)Unmap6: 10568: datastore1: Unmap capability 0 OP i: 0 offset: 65536 endROffset: 589824 Acquired: TRUE
[YYYY-MM-DDTHH:MM:SS]3.068Z cpu60:2098008)WARNING: HBX: 2723: Failed to cleanup registration key on volume6849748c-efa69ae7-7246-5c2573e0e3c2: FailureESC[0m
[YYYY-MM-DDTHH:MM:SS] cpu60:2098008)WARNING: Vol3: 4678: Error closing the volume: . Eviction fails: FailureESC[0m
[YYYY-MM-DDTHH:MM:SS] cpu56:3623345)SchedVsi: 2208: Group: host/opt/vsan/vsanperfsvc(2756331): min=0, units: mb
[YYYY-MM-DDTHH:MM:SS] cpu74:2098732)Unmap6: 10536: datastore1: Acquired UC with 0 OP ucOffset 69632 ucIndex: 2
[YYYY-MM-DDTHH:MM:SS] cpu74:2098732)Unmap6: 10568: datastore1: Unmap capability 0 OP i: 0 offset: 65536 endROffset: 589824 Acquired: TRUE
[YYYY-MM-DDTHH:MM:SS] cpu58:2098008)WARNING: HBX: 2723: Failed to cleanup registration key on volume6849748c-efa69ae7-7246-5c2573e0e3c2: FailureESC[0m
[YYYY-MM-DDTHH:MM:SS] cpu58:2098008)WARNING: Vol3: 4678: Error closing the volume: . Eviction fails: FailureESC[0m
[YYYY-MM-DDTHH:MM:SS] cpu122:2100149)FSS: 7434: Failed to open file 'hpilo-d0ccb9'; Requested flags 0x5, world: 2100149 [sut], (Existing flags 0x5, world: 2101011 [ahsdv]): Busy
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)World: 3355: PRDA 0x42004c800000 ss 0x0 ds 0x103 es 0x103 fs 0x103 gs 0x0
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)World: 3357: TR 0x758 GDT 0x453940352888 (0xffff) IDT 0x42002d0a3000 (0xffff)
[YYYY-MM-DDTHH:MM:SS] cpu50:2097202)World: 3359: CR0 0x80010031 CR3 0x8102907000 CR4 0x140768


Environment

VMware vSphere ESXi 8.x

Cause

This issue occurs due to a possible conflict between the iLO Driver with other software that may be causing interruption with VMware ESXi 8.x and the AMD processor.

 

Resolution

This is a issue with Hardware and not related to VMware vSphere ESXi.

Below is the resolution provided by HPE.

To resolve this issue, perform the following steps:

1. Go to System Configuration.
2. Go to the BIOS/Platform Configuration (RBSU).
3. Go to "Workload Profile" and select "Virtualization - Max Performance".
   
   
   
   
4. Go to "Power and Performance Options" and select the following options:
• For Power Regulator: Select "Static High Performance Mode".
• For Minimum Processor Idle Power Core C-State: Select "No C-states".
• For Collaborative Power Control: Select "Disable".
  
  
  
  
  

If you need any further assistance, please contact HPE.