PSOD : NMI IPI: Panic requested by another PCPU. RIPOFF(base):RBP:CS [0x1e0276(0x420004600000):0x453a5921f000:0xf48] (Src 0x1, CPU27)

Products

• VMware vSphere ESXi

Issue/Introduction

• vCenter version being 7.0u3o, after patching, the PSOD was observed.
• Presence of zdriver (zerto driver)
• Stack Trace is as below:

#0  Panic_WithBacktrace (sbt=sbt@entry=0x4303ba005118, fmt=fmt@entry=0x420004bf3e68 "NMI IPI: Panic requested by another PCPU. RIPOFF(base):RBP:CS [%#lx(%#lx):%#lx:%#lx] (Src %#x, CPU%u)") at bora/vmkernel/main/panic.c:147
#1  0x00004200046fbce9 in NMIHandleBtOrHaltRequest (source=NMI_SRC_HEARTBEAT, fullFrame=0x4529c01b2f40) at bora/vmkernel/main/nmi.c:699
#2  NMIHandleIPISource (fullFrame=0x4529c01b2f40) at bora/vmkernel/main/nmi.c:534
#3  NMI_Interrupt (fullFrame=fullFrame@entry=0x4529c01b2f40) at bora/vmkernel/main/nmi.c:753
#4  0x0000420004753393 in IDTNMIWork (fullFrame=0x4529c01b2f40) at bora/vmkernel/main/x86/idt.c:1495
#5  0x00004200047546dd in Int2_NMI (fullFrame=0x4529c01b2f40) at bora/vmkernel/main/x86/idt.c:760
#6  0x000042000474e068 in gate_entry ()
#7  0x00004200047e0276 in VPMem_PPN2BPN (world=0x453a5921f000, ppn=274880466480, numPages=1, nextRangePPN=0x453a3b91b338) at bora/vmkernel/mem/vpmem.c:2592
#8  0x00004200047b8b72 in VmMemPin_ReleasePhysMemRange (world=world@entry=0x453a5921f000, startGuestPPN=startGuestPPN@entry=274880466480, numPages=numPages@entry=1, source=source@entry=VMMEMPIN_VMK) at bora/vmkernel/mem/vmmempin.c:1153
#9  0x000042000477448c in P2MCache_Release (world=world@entry=0x453a5921f000, ppn=ppn@entry=274880466480) at bora/vmkernel/mem/p2mcache.c:519
#10 0x000042000477e0d1 in PhysMem_ReleaseSGE (world=world@entry=0x453a5921f000, tppn=274880466480, fireTraces=fireTraces@entry=0 '\000') at bora/vmkernel/mem/physMem_vmk.c:2765
#11 0x0000420004b54298 in VSCSI_UnpinPages (world=world@entry=0x453a5921f000, sgList=sgList@entry=0x45da7c8abf78, fireWriteTraces=0 '\000') at bora/vmkernel/virtscsi/vscsi_hba.c:654
#12 0x0000420004b71e53 in LSIProcessCompletionInt (world=0x453a5921f000, entry=<optimized out>, handle=<optimized out>, vmmData=<optimized out>, outResult=<optimized out>, replySG=0x45da7c8abf78, monActionStatus=VMK_OK, canBlock=0 '\000') at bora/vmkernel/virtscsi/lsi_vmkdev.c:1006
#13 0x0000420004b7218d in LSIDoIOComplete (info=<optimized out>, world=0x453a5921f000, handle=0x430eab67bb40, token=0x45da51ac85c0, vcpuInitiatorHint=<optimized out>) at bora/vmkernel/virtscsi/lsi_vmkdev.c:1438
#14 0x0000420004b4b7f3 in VSCSIDoIOComplete (vcpuInitiator=0x453a3b91b53c, token=0x45da51ac85c0, handle=0x430eab67bb40, world=0x453a5921f000) at bora/vmkernel/virtscsi/vscsi.c:5914
#15 VSCSIPostIOCompletion (handle=0x430eab67bb40, token=0x45da51ac85c0) at bora/vmkernel/virtscsi/vscsi.c:6185
#16 0x00004200046be371 in AsyncPopCallbackFrameInt (token=0x45da51ac85c0, runCallback=<optimized out>, wantCallback=<optimized out>, ra=<optimized out>) at bora/vmkernel/main/async_io.c:575
#17 0x0000420005deea82 in ?? ()
#18 0x0000431200000001 in ?? ()
#19 0x0000000000000000 in ?? ()

Environment

VMware vSphere ESXi 7.0.3

Cause

The PSOD happened because the thread handled invalid PPN (0x4000270e30) and that caused the thread to be in a infinite loop. The invalid PPN was coming from VSCSI_SGPinArrType data which was used by VSCSI layer. The valid PPN was 0x270e30 and the wrong PPN  being the result of 1-bit-flip on bit38 of 0x270e30. This was an issue on processor/ucode or somewhere on the hardware platform causing 1 bit flip on data.

The stack backtrace involves zdriver. It is from Zerto solution and it is supported by the vendor.

Resolution

• Recommended to run hardware diagnostic and stress tests to check the hardware and updating BIOS/ucode to fix the ucode.
• Contact Zerto who created and supports zdriver.