Host crashes with PSOD- NMI IPI: Panic requested by another PCPU.

Products

• VMware vSphere ESXi

Issue/Introduction

Host crashes with PSOD- NMI IPI: Panic requested by another PCPU.

VMware ESXi 7.0.3 [Releasebuild-24784741 x86_64]
NMI IPI: Panic requested by another PCPU. RIPOFF (base):RBP:CS [0x109176(0x42002ba00000):0x3ff :0xf48] (Src 0x1, CPU65)
cr0=0x8001003d cr2=0xa11858b667 cr3=0x5eef7000 cr4=0x10216c
FMS=06/6a/6 uCode=0xd0003f5
×PCPU65:2099116/vmk3-rx-2
PCPU 0: SSSUSSSSSSSSSSSSSSSSSSSSSSSSSSISSSSUUSUSSSSS IS IUSISSUUSSSSUSISUS
PCPU 64: SSSSSSSSSSSSUSSSSSSUSUSSUSSSISSSSSSSUUSSSSUUUSSSUSSSSSSSSUUSSSUS
Code start: 0x42002ba00000 VMK uptine: 1:18:12:23.431
Saved backtrace from: pcpu 65 Heartbeat NMI
0x453caea9bc30 : [0x42002bb09175]SP_WaitLock@vmkernel#nover+0xc2 stack: 0x431cecc37a08
0x453caca9bc70: [0x42002bb09274]SPLockWork@vnkernel#nover+0x29 stack: 0x431cecc37a08
8x153caca9bc80:[0x42002bb09426]SPAcqHriteLockWork@vmkerncl#nover+0x3b stack: 0x41ffd240b760
0x453caea9bca0: [0x42002cfc1914]rw_wlock@(tcpip4)#<None>+0x8d stack: 0x453c82d9f140
0x453caea9bcb0: [0x42002d01d149]tcp_input@(tcpip4)#<None>+0x1f6 stack: 0x45dcfa7ba700
ax453caca9bdf:[0x42002cfc334e]netisr_dispatch@(tcpip4)#<Nonc>+0x17 stack: 0xef
0x453caea9be00: [0x42002cfec50b]ether_demux@(tcpip4)#<None>+0x1f8 stack: 0x431cef 162b90
0x453caea9be30: [0x42002cfec77f]ether_input@(tcpip4)#<None>+0x194 stack: 0x1
0x453caca9be70:[0x42002cfbd5ff]if_vnk_rx@(tcpip4)#<None>+x1bc stack: 0x202
0x453caca9bef0: [0x42002cfac038]TcpipRxFastPath@(tcpip4)#<None>+0x71 stack: 0x453caca9bf80
0x453caea9bf 20 : [0x42002cfac2e3]TcpipRx@(tcpip4)#<None>+0x144 stack: 0x0
0x453caea9bf80: [0x42002cfa9185]TcpipDispatchWor 1d@(tcpip4)#<None>+0x186 stack: 0x45dd0b169bc0
Bx453caca9bfe0: [0x42002bdb4cbd]CpuSched_StartWor ld@vmkernel#nover+0x86 stack: 0x0
0x453caea9c000: [0x42002bac4fff ]Debug_IsInitialized@vmkernel#nover+Oxc stack: 0x0
base fs=0x0 gs=0x420050400000 Kgs=0x0
l other PCPU is in panic.
XXXX-XX-XXT07:42:12.911Z cpu97:2099115)NMI: 712: NMI IPI: RIPOFF(base) :RBP:CS [0x1091al(0x42002ba00000):0x3ff :0xf48] (Src 0x1, C
PU97)
XXXX-XX-XXT07:42:39.999Z cpu119:2098510)NMI: 712: NMI IPI: RIPOFF (base) :RBP:CS [0x85c56(0x42002ba00000):0x1:0xf48] (Src 0x4, CPU
119)
XXXX-XX-XXT07:42:31.911Z cpu117:2099117)NMI: 712: NMI IPI: RIPOFF(base):RBP:CS [0x1091a1(0x42002ba00000):0x3ff :0xf48] (Src 0x1,
CPU117)
XXXX-XX-XXT07:42:28.912Z cpu67:2099112)NMI: 712: NMI IPI: RIPOFF(base):RBP:CS [0x1091a1(0x42002ba00000):0x3ff:0xf48] (Src 0x1, C
PU67)
XXXX-XX-XXT07:42:26.901Z cpu97:2099115)NMI: 712: NMI IPI: RIPOFF(base) :RBP:CS [0x109176(0x42002ba00000):0x3ff :0xf48] (Src 0x1, C
PU97)
XXXX-XX-XXT07:42:22.929Z cpu61:2099114)NMI: 712: NMI IPI: RIPOFF(basc):RBP:CS [0x143b87(0x42002ba00000):0x20:0xf48] (Src 0x1, CP
U61)
XXXX-XX-XXT07:42:18.930Z cpu31:2099111)NMI: 712: NMI IPI: RIPOFF(base) :RBP:CS [0x109176(0x42002ba00000):0x3ff:0xf48] (Src 0x1, C
PU31)
XXXX-XX-XXT07:42:17.919Z cpu78:2099110)NMI: 712: NMI IPI: RIPOFF(base) :RBP:CS [0x1091a1(0x42002ba00000):0x3ff :0xf48] (Src 0x1, C
PU78)
XXXX-XX-XXT07:42:17.910Z cpu117:2099117)NMI: 712: NMI IPI: RIPOFF (base):RBP:CS [0x1091a1(0x42002ba00000):0x3ff:0xf48] (Src 0x1,
No place on disk to dunp data.
VASpace (08/15)

 

/var/log/vmkernel.log:

XXXX-XX-XXT01:47:49.712Z cpu48:2099111)0x453cacc9bfe0:[0x42002bdb4ebd]CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0
XXXX-XX-XXT01:47:49.714Z cpu48:2099111)0x453cacc9c000:[0x42002bac4fff]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0
XXXX-XX-XXT01:47:50.037Z cpu119:2098656)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x28 (0x45bcc2381648, 0) to dev "eui.5c8287dc00bf6a0f2016171f000016a3" on path "vmhba64:C0:T22:L163" Failed:
XXXX-XX-XXT01:47:50.044Z cpu119:2098656)NMP: nmp_ThrottleLogForDevice:3875: H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x0 0x0. Act:NONE. cmdId.initiator=0x430cccc14040 CmdSN 0x44063
XXXX-XX-XXT01:47:50.056Z cpu119:2098656)ScsiDeviceIO: 4173: Cmd(0x45bcc2381648) 0x28, CmdSN 0x44063 from world 0 to dev "eui.5c8287dc00bf6a0f2016171f000016a3" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x0 0x0
XXXX-XX-XXT01:47:50.260Z cpu50:2098791)HBX: 747: Reading HB at 3407872 on vol 'SIO-APCMNTVSIMVGPVMG01-03' failed: I/O error
XXXX-XX-XXT01:47:50.263Z cpu50:2098791)WARNING: HBX: 3685: 'SIO-APCMNTVSIMVGPVMG01-03': HB at offset 3407872 - Reclaiming timed out HB failed: I/O error:
XXXX-XX-XXT01:47:50.263Z cpu50:2098791)WARNING:   [HB state abcdef02 offset 3407872 gen 205 stampUS 130554611532 uuid 694401d9-3d6037d1-76ee-a088c24f438a jrnl <FB 25165824> drv 24.82 lockImpl 4 ip 10.155.16.12]
XXXX-XX-XXT01:47:50.606Z cpu72:2097799)WARNING: Heartbeat: 827: PCPU 122 didn't have a heartbeat for 21 seconds, timeout is 14, 2 IPIs sent; *may* be locked up.
XXXX-XX-XXT01:47:50.611Z cpu122:2099114)ALERT: NMI: 712: NMI IPI: RIPOFF(base):RBP:CS [0x10912b(0x42002ba00000):0x3ff:0xf48] (Src 0x1, CPU122)
XXXX-XX-XXT01:47:50.611Z cpu47:2097774)WARNING: Heartbeat: 827: PCPU 98 didn't have a heartbeat for 21 seconds, timeout is 14, 2 IPIs sent; *may* be locked up.
XXXX-XX-XXT01:47:50.624Z cpu98:2099117)ALERT: NMI: 712: NMI IPI: RIPOFF(base):RBP:CS [0x109176(0x42002ba00000):0x3ff:0xf48] (Src 0x1, CPU98)
XXXX-XX-XXT01:47:50.633Z cpu122:2099114)0x453caec9bc30:[0x42002bb0912a]SP_WaitLock@vmkernel#nover+0x77 stack: 0x431cecc37a08
XXXX-XX-XXT01:47:50.637Z cpu122:2099114)0x453caec9bc70:[0x42002bb09274]SPLockWork@vmkernel#nover+0x29 stack: 0x431cecc37a08
XXXX-XX-XXT01:47:50.639Z cpu122:2099114)0x453caec9bc80:[0x42002bb09426]SPAcqWriteLockWork@vmkernel#nover+0x3b stack: 0x41ffd240b760
XXXX-XX-XXT01:47:50.639Z cpu98:2099117)0x453cae89bc30:[0x42002bb09175]SP_WaitLock@vmkernel#nover+0xc2 stack: 0x431cecc37a08
XXXX-XX-XXT01:47:50.639Z cpu122:2099114)0x453caec9bca0:[0x42002cfc1914]rw_wlock@(tcpip4)#<None>+0x8d stack: 0x453caec9f140
XXXX-XX-XXT01:47:50.642Z cpu122:2099114)0x453caec9bcb0:[0x42002d01d149]tcp_input@(tcpip4)#<None>+0x1f6 stack: 0x45dd04e39600
XXXX-XX-XXT01:47:50.642Z cpu98:2099117)0x453cae89bc70:[0x42002bb09274]SPLockWork@vmkernel#nover+0x29 stack: 0x431cecc37a08
XXXX-XX-XXT01:47:50.642Z cpu122:2099114)0x453caec9bdf0:[0x42002cfc334e]netisr_dispatch@(tcpip4)#<None>+0x17 stack: 0x4200000000ef
XXXX-XX-XXT01:47:50.647Z cpu98:2099117)0x453cae89bc80:[0x42002bb09426]SPAcqWriteLockWork@vmkernel#nover+0x3b stack: 0x41ffd240b760
XXXX-XX-XXT01:47:50.647Z cpu122:2099114)0x453caec9be00:[0x42002cfec50b]ether_demux@(tcpip4)#<None>+0x1f8 stack: 0x431cef162b90
XXXX-XX-XXT01:47:50.651Z cpu98:2099117)0x453cae89bca0:[0x42002cfc1914]rw_wlock@(tcpip4)#<None>+0x8d stack: 0x453cae89f140
XXXX-XX-XXT01:47:50.651Z cpu122:2099114)0x453caec9be30:[0x42002cfec77f]ether_input@(tcpip4)#<None>+0x194 stack: 0x1
XXXX-XX-XXT01:47:50.652Z cpu98:2099117)0x453cae89bcb0:[0x42002d01d149]tcp_input@(tcpip4)#<None>+0x1f6 stack: 0x45dcf1417800
XXXX-XX-XXT01:47:50.652Z cpu122:2099114)0x453caec9be70:[0x42002cfbd5ff]if_vmk_rx@(tcpip4)#<None>+0x1bc stack: 0x202
XXXX-XX-XXT01:47:50.652Z cpu98:2099117)0x453cae89bdf0:[0x42002cfc334e]netisr_dispatch@(tcpip4)#<None>+0x17 stack: 0x4200000000ef
XXXX-XX-XXT01:47:50.652Z cpu122:2099114)0x453caec9bef0:[0x42002cfac038]TcpipRxFastPath@(tcpip4)#<None>+0x71 stack: 0x453caec9bf80
XXXX-XX-XXT01:47:50.657Z cpu98:2099117)0x453cae89be00:[0x42002cfec50b]ether_demux@(tcpip4)#<None>+0x1f8 stack: 0x431cef162b90
XXXX-XX-XXT01:47:50.662Z cpu122:2099114)0x453caec9bf20:[0x42002cfac2e3]TcpipRx@(tcpip4)#<None>+0x144 stack: 0x0
XXXX-XX-XXT01:47:50.662Z cpu98:2099117)0x453cae89be30:[0x42002cfec77f]ether_input@(tcpip4)#<None>+0x194 stack: 0x1
XXXX-XX-XXT01:47:50.663Z cpu122:2099114)0x453caec9bf80:[0x42002cfa9185]TcpipDispatchWorld@(tcpip4)#<None>+0x186 stack: 0x45bd36c7bb00
XXXX-XX-XXT01:47:50.663Z cpu98:2099117)0x453cae89be70:[0x42002cfbd5ff]if_vmk_rx@(tcpip4)#<None>+0x1bc stack: 0x212
XXXX-XX-XXT01:47:50.663Z cpu122:2099114)0x453caec9bfe0:[0x42002bdb4ebd]CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0
XXXX-XX-XXT01:47:50.663Z cpu98:2099117)0x453cae89bef0:[0x42002cfac038]TcpipRxFastPath@(tcpip4)#<None>+0x71 stack: 0x453cae89bf80
XXXX-XX-XXT01:47:50.663Z cpu122:2099114)0x453caec9c000:[0x42002bac4fff]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0
XXXX-XX-XXT01:47:50.663Z cpu98:2099117)0x453cae89bf20:[0x42002cfac2e3]TcpipRx@(tcpip4)#<None>+0x144 stack: 0x0
XXXX-XX-XXT01:47:50.673Z cpu98:2099117)0x453cae89bf80:[0x42002cfa9185]TcpipDispatchWorld@(tcpip4)#<None>+0x186 stack: 0x45bd1fe71180
XXXX-XX-XXT01:47:50.677Z cpu98:2099117)0x453cae89bfe0:[0x42002bdb4ebd]CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0
XXXX-XX-XXT01:47:50.679Z cpu98:2099117)0x453cae89c000:[0x42002bac4fff]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0
XXXX-XX-XXT01:47:50.771Z cpu118:2098500)WARNING: PowerFlex mapVolIO_Do_CK:1467 :Mit: 0x45dd04d3c600. Retrying IO Type READ. Failed comb: 6a5d000f8015. offsetInComb 15237632. SizeInLB 8. SDS_ID cbcb150d00000025. Comb Gen 2c. Head Gen 4ac.
XXXX-XX-XXT01:47:50.851Z cpu72:2098500)WARNING: PowerFlex mapVolIO_Do_CK:1481 :Mit: 0x45dd04d3c600. Vol ID 0x201617160000169a. Last fault Status SUCCESS(65).Last error Status NOT_CONN(4) Retry count (9) chan (3)
XXXX-XX-XXT01:47:54.594Z cpu91:2098656)ScsiDeviceIO: 4173: Cmd(0x45dcfb66cdc8) 0x28, CmdSN 0x4401e from world 0 to dev "eui.5c8287dc00bf6a0f2016171e000016a2" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x0 0x0
XXXX-XX-XXT01:47:54.605Z cpu79:2098500)WARNING: PowerFlex mapVolIO_Do_CK:1481 :Mit: 0x45bcd3b7cb40. Vol ID 0x2016171c000016a0. Last fault Status SUCCESS(65).Last error Status NOT_CONN(4) Retry count (1) chan (3)
XXXX-XX-XXT01:47:54.695Z cpu91:2098656)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x28 (0x45bd43933dc8, 0) to dev "eui.5c8287dc00bf6a0f20161721000016a5" on path "vmhba64:C0:T22:L165" Failed:
XXXX-XX-XXT01:47:54.699Z cpu91:2098656)NMP: nmp_ThrottleLogForDevice:3875: H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x0 0x0. Act:NONE. cmdId.initiator=0x430cccdbbf80 CmdSN 0x1e6a6
XXXX-XX-XXT01:47:54.695Z cpu92:2098500)WARNING: PowerFlex mapVolIO_Do_CK:1467 :Mit: 0x45bd41fc6d40. Retrying IO Type READ. Failed comb: 6a63006f8015. offsetInComb 1376768. SizeInLB 8. SDS_ID cbcb150d00000025. Comb Gen 1. Head Gen 500.
XXXX-XX-XXT01:47:54.706Z cpu91:2098656)ScsiDeviceIO: 4173: Cmd(0x45bd43933dc8) 0x28, CmdSN 0x1e6a6 from world 0 to dev "eui.5c8287dc00bf6a0f20161721000016a5" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x0 0x0
XXXX-XX-XXT01:47:54.703Z cpu92:2098500)WARNING: PowerFlex mapVolIO_Do_CK:1481 :Mit: 0x45bd41fc6d40. Vol ID 0x2016171f000016a3. Last fault Status SUCCESS(65).Last error Status NOT_CONN(4) Retry count (2) chan (3)
XXXX-XX-XXT01:47:54.726Z cpu107:2098792)HBX: 747: Reading HB at 3407872 on vol 'SIO-APCMNTVSIMVGPVMG01-04' failed: I/O error
XXXX-XX-XXT01:47:54.733Z cpu107:2098792)WARNING: HBX: 3685: 'SIO-APCMNTVSIMVGPVMG01-04': HB at offset 3407872 - Reclaiming timed out HB failed: I/O error:
XXXX-XX-XXT01:47:54.737Z cpu107:2098792)WARNING:   [HB state abcdef02 offset 3407872 gen 207 stampUS 130554611557 uuid 694401d9-3d6037d1-76ee-a088c24f438a jrnl <FB 8388608> drv 24.82 lockImpl 4 ip 10.155.16.12]
XXXX-XX-XXT01:47:54.793Z cpu27:2674794)HBX: 747: Reading HB at 3407872 on vol 'SIO-APCMNTVSIMVGPVMG01-09' failed: I/O error
XXXX-XX-XXT01:47:54.801Z cpu27:2674794)WARNING: HBX: 3685: 'SIO-APCMNTVSIMVGPVMG01-09': HB at offset 3407872 - Reclaiming timed out HB failed: I/O error:
XXXX-XX-XXT01:47:54.801Z cpu27:2674794)WARNING:   [HB state abcdef02 offset 3407872 gen 179 stampUS 130554611693 uuid 694401d9-3d6037d1-76ee-a088c24f438a jrnl <FB 16777216> drv 24.82 lockImpl 4 ip 10.155.16.12]
XXXX-XX-XXT01:47:56.116Z cpu94:2098500)WARNING: PowerFlex mapVolIO_Do_CK:1467 :Mit: 0x45dd04d3c600. Retrying IO Type READ. Failed comb: 6a5d000f8015. offsetInComb 15237632. SizeInLB 8. SDS_ID cbcb150d00000025. Comb Gen 2c. Head Gen 4ac.
XXXX-XX-XXT01:47:56.171Z cpu94:2098500)WARNING: PowerFlex mapVolIO_Do_CK:1481 :Mit: 0x45dd04d3c600. Vol ID 0x201617160000169a. Last fault Status SUCCESS(65).Last error Status NOT_CONN(4) Retry count (12) chan (3)
XXXX-XX-XXT01:47:56.231Z cpu64:2098500)WARNING: PowerFlex mapVolIO_Do_CK:1467 :Mit: 0x45bcd6e3c7c0. Retrying IO Type READ. Failed comb: 6a5f80648015. offsetInComb 8372736. SizeInLB 8. SDS_ID cbcb150e00000026. Comb Gen 4cb. Head Gen 4d2.
XXXX-XX-XXT01:47:56.251Z cpu51:2098501)WARNING: PowerFlex mapVolIO_Do_CK:1467 :Mit: 0x45dcfb4ee400. Retrying IO Type READ. Failed comb: 6a6080658015. offsetInComb 11026944. SizeInLB 8. SDS_ID cbcb150a0000001d. Comb Gen 4cd. Head Gen 4d4.
XXXX-XX-XXT01:47:56.246Z cpu64:2098500)WARNING: PowerFlex mapVolIO_Do_CK:1481 :Mit: 0x45bcd6e3c7c0. Vol ID 0x2016171a0000169e. Last fault Status SUCCESS(65).Last error Status NOT_CONN(4) Retry count (19) chan (3)
XXXX-XX-XXT01:47:56.258Z cpu64:2098500)WARNING: PowerFlex mapVolIO_Do_CK:1467 :Mit: 0x45dd04c98f80. Retrying IO Type WRITE. Failed comb: 6a64007081d3. offsetInComb 12832776. SizeInLB 1. SDS_ID cbcb150f00000034. Comb Gen 1. Head Gen 4f7.
XXXX-XX-XXT01:47:56.262Z cpu64:2098500)WARNING: PowerFlex mapVolIO_Do_CK:1481 :Mit: 0x45dd04c98f80. Vol ID 0x20161720000016a4. Last fault Status SUCCESS(65).Last error Status NOT_CONN(4) Retry count (8) chan (3)

Environment

VMware ESXi Version: 7.0.3

Cause

Non-Maskable Interrupt (NMI) detected in system traces, indicating a potential hardware failure.

Resolution

The PCPU has become unresponsive, accompanied by a Non-Maskable Interrupt (NMI)—a pattern commonly indicative of an underlying hardware issue.

Engage the hardware vendor for a thorough investigation, as this may point to a failing CPU or motherboard component that could require diagnostics or replacement.