Manual Crash Triggered by NMI on Virtual machine hosted on vCenter/ESXi
search cancel

Manual Crash Triggered by NMI on Virtual machine hosted on vCenter/ESXi

book

Article ID: 389912

calendar_today

Updated On:

Products

VMware vSphere ESX 8.x VMware vSphere ESXi 8.0

Issue/Introduction

  • A manual crash was triggered on the system using a Non-Maskable Interrupt (NMI), as indicated by the dump analysis from the OS vendor.

NMI_HARDWARE_FAILURE (80)
This is typically due to a hardware malfunction.  The hardware supplier should
be called.
Arguments:
Arg1: 00000000004f4454, 'TDO'
Arg2: 0000000000000000, Status Byte
Arg3: 0000000000000000
Arg4: 0000000000000000

Debugging Details:
VIRTUAL_MACHINE:  VMware
FAULTING_THREAD:  fffff8021b1c5980
PROCESS_NAME:  System
STACK_TEXT: 
fffff802`1d02fc08 fffff802`1b65aa4e     : 00000000`00000080 00000000`004f4454 00000000`00000000 00000000`00000000 : nt!KeBugCheckEx [d:\rs1\minkernel\ntos\ke\amd64\procstat.asm @ 127]
fffff802`1d02fc10 fffff802`1b034920     : ffff810c`6aa82968 fffff802`1b672e30 fffff802`1b672e30 00000000`00000001 : hal!HalBugCheckSystem+0x7e [d:\rs1\minkernel\hals\lib\whea\mca.c @ 3171]
fffff802`1d02fc50 fffff802`1b65ba1e     : fffff802`000006c0 fffff802`1b1c3630 fffff802`1d02fd30 fffff802`1aec41b8 : nt!WheaReportHwError+0x258 [d:\rs1\minkernel\ntos\whea\whea.c @ 810]
fffff802`1d02fcb0 fffff802`1aec3ad6     : 00000000`00000001 00000000`00000000 000000a8`1f6b2685 fffff802`1aec3f88 : hal!HalHandleNMI+0xfe [d:\rs1\minkernel\hals\lib\whea\nmi.c @ 396]
fffff802`1d02fce0 fffff802`1af6e242     : 00000000`00000001 fffff802`1d02fef0 00000000`00000000 00000000`00000000 : nt!KiProcessNMI+0x106 [d:\rs1\minkernel\ntos\ke\amd64\misc.c @ 298]
fffff802`1d02fd30 fffff802`1af6e043     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KxNmiInterrupt+0x82 [d:\rs1\minkernel\ntos\ke\amd64\trap.asm @ 697]
fffff802`1d02fe70 fffff802`1b011cd9     : 00000000`00000046 ffff810c`6ac460f0 00000000`00000000 00000000`00000000 : nt!KiNmiInterrupt+0x1c3 [d:\rs1\minkernel\ntos\ke\amd64\trap.asm @ 657]
fffff802`1d01c200 fffff802`1ae66c8e     : fffff802`1b011cf0 00000000`00000000 ffff810c`6ac46010 00000000`000000ae : nt!PpmIdleGuestExecute+0x15 [d:\rs1\minkernel\ntos\po\ppmhv.c @ 697]
fffff802`1d01c240 fffff802`1ae65e1a     : fffff804`0fa65f40 012d92f8`012d92f8 00000000`00000000 00000000`00000000 : nt!PpmIdleExecuteTransition+0xcbe [d:\rs1\minkernel\ntos\po\ppmidle.c @ 3925]
fffff802`1d01c4c0 fffff802`1af6611c     : 00000000`00000000 fffff802`1b149180 fffff802`1b1c5980 ffff810c`79728040 : nt!PoIdle+0x33a [d:\rs1\minkernel\ntos\po\ppmidle.c @ 1046]
fffff802`1d01c620 00000000`00000000     : fffff802`1d01d000 fffff802`1d016000 00000000`00000000 00000000`00000000 : nt!KiIdleLoop+0x2c [d:\rs1\minkernel\ntos\ke\amd64\idle.asm @ 110]

  • VM found in a suspended state without any information in the tasks.
  • May encounter a VM in a suspended state, with the task not providing sufficient information.
  • Users may assert that no suspended tasks were initiated by them.

 

Environment

VMware vSphere ESX 8.x

VMware vSphere ESX 7.x

Cause

  • In this scenario, you might encounter a virtual machine (VM) in a suspended state, with the task not providing sufficient information. Users may assert that no suspended tasks were initiated by them, suggesting that the VM's suspension was not a result of user intervention.
  • Error reported from VMware log and hostd.logs shows that VM crashed. The VM's failure to suspend automatically led to a manual crash via NMI to address operational inefficiencies.

/var/run/log/hostd.log

YYYY-MM-DD TT:37.090Z info hostd[2162299] [Originator@6876 sub=Vimsvc.CgiServiceManager opID=CGI-server-8b44] Validating CGI ticket for URL '/cgi-bin/vm-support.cgi?listmanifests=1'
YYYY-MM-DD TT:51.170Z info hostd[2162204] [Originator@6876 sub=Vimsvc.CgiServiceManager opID=CGI-server-8b5b] Validating CGI ticket for URL '/cgi-bin/vm-support.cgi?manifests=VirtualMachines:CoreDumpHung HungVM:Send_NMI_To_Guest HungVM:Suspend_VM'
YYYY-MM-DD:45.908Z info hostd[2162334] [Originator@6876 sub=Vimsvc.CgiServiceManager opID=CGI-server-9088] Validating CGI ticket for URL '/cgi-bin/vm-support.cgi?manifests=VirtualMachines:CoreDumpHung'

/vmfs/volume/datastore/<VM>/vmware.log

YYYY-MM-DD TT.499Z Wa(03) vcpu-0 - WinBSOD: Synthetic MSR[0x40000100] 0x80
YYYY-MM-DD TT.499Z Wa(03)+ vcpu-0 -
YYYY-MM-DD TT.499Z Wa(03) vcpu-0 - WinBSOD: Synthetic MSR[0x40000101] 0x4f4454
YYYY-MM-DD TT.499Z Wa(03)+ vcpu-0 -
YYYY-MM-DD TT.499Z Wa(03) vcpu-0 - WinBSOD: Synthetic MSR[0x40000102] 0x0
YYYY-MM-DD TT.499Z Wa(03)+ vcpu-0 -
YYYY-MM-DD TT.499Z Wa(03) vcpu-0 - WinBSOD: Synthetic MSR[0x40000103] 0x0
YYYY-MM-DD TT.499Z Wa(03)+ vcpu-0 -
YYYY-MM-DD TT.499Z Wa(03) vcpu-0 - WinBSOD: Synthetic MSR[0x40000104] 0x0
YYYY-MM-DD TT.499Z Wa(03)+ vcpu-0 -YYYY-MM-DD TT.521Z In(05) vmx - SUSPEND: Start suspend (flags=0)
YYYY-MM-DD TT.535Z In(05) vcpu-0 - Progress -1% (msg.checkpoint.saveStatus)
YYYY-MM-DD TT.535Z In(05) vcpu-0 - Checkpointed in VMware ESX, 7.0.3, build-22348816, Linux Host
YYYY-MM-DD TT.679Z In(05) vcpu-0 - Progress 0% (none)
YYYY-MM-DD TT.679Z In(05) vcpu-0 - MainMem: Writing full memory image, '/vmfs/volumes/XXXXXXXX-XXXXXXXX-XXX-XXXXXXXXXX/VMName/VMName-XXXXXXXX.vmem'.
YYYY-MM-DD TT:53.011Z In(05) vcpu-0 - Progress 1% (none)
YYYY-MM-DD TT:54.394Z In(05) vcpu-0 - Progress 2% (none)
YYYY-MM-DD TT:55.768Z In(05) vcpu-0 - Progress 3% (none)
YYYY-MM-DD TT:57.143Z In(05) vcpu-0 - Progress 4% (none)
YYYY-MM-DD TT:14.637Z In(05) vcpu-0 - SUSPEND: Completed suspend: 'Operation completed successfully' (0)

YYYY-MM-DD TT:14.637Z In(05) vmx - Stopping VCPU threads...
YYYY-MM-DD TT:14.637Z In(05) vcpu-0 - VMMon_WaitForExit: vcpu-0: worldID=2103623
YYYY-MM-DD TT:14.637Z In(05) vcpu-1 - VMMon_WaitForExit: vcpu-1: worldID=2103627
YYYY-MM-DD TT:14.637Z In(05) vcpu-3 - VMMon_WaitForExit: vcpu-3: worldID=2103629
YYYY-MM-DD TT:14.637Z In(05) vcpu-2 - VMMon_WaitForExit: vcpu-2: worldID=2103628
YYYY-MM-DD TT:14.637Z In(05) vcpu-4 - VMMon_WaitForExit: vcpu-4: worldID=2103630
YYYY-MM-DD TT:14.637Z In(05) vmx - MonitorVMMCoreRequest: ********************************************
YYYY-MM-DD TT:14.637Z In(05) vmx - MonitorVMMCoreRequest: Sync core dump requested; not a real fault
YYYY-MM-DD TT:14.637Z In(05) vmx - MonitorVMMCoreRequest: ********************************************
YYYY-MM-DD TT:14.637Z In(05) vcpu-5 - VMMon_WaitForExit: vcpu-5: worldID=2103631
YYYY-MM-DD TT:14.637Z Wa(03) vmx - Dumping vmx core per request
YYYY-MM-DD TT:15.658Z Wa(03) vmx - A core file is available in "/vmfs/volumes/XXXXXXXX-XXXXXXXX-XXX-XXXXXXXXXX/VMName-XXXXXXX/vmx-zdump.000"

Resolution

When user initiated a log bundle generation by selecting the HungVM option, causing the VM to enter a suspend state. The traces indicate that there are no underlying issues with either ESXi or vCenter. 

Additional Information

Powered by Wolken Software