Virtual Machine powered off with error: <vm name> contained the host physical page ### which was scheduled for immediate retirement.
search cancel

Virtual Machine powered off with error: <vm name> contained the host physical page ### which was scheduled for immediate retirement.

book

Article ID: 404366

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  •  Virtual machines experience an unexpected or ungraceful power-off event.

  • High Availability (HA) failover is initiated, automatically restarting the virtual machines on an available host.

  • Events display messages like:

Description: <vm name> contained the host physical page ### which was scheduled for immediate retirement. To avoid system instability the virtual machine is forcefully powered off.

Event ID: esx.problem.vm.kill.unexpected.forcefulPageRetire.64.2

  • Comparable entries can be found within the host daemon log at /var/run/log/hostd.log:
    YYYY-MM-DDT HH:MM:SS In(166) Hostd[2099322]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/########-########-####-############/##-###-#####/##->###-#####.vmx] Deferring power on completion until VM is at a stable state
YYYY-MM-DDT HH:MM:SS In(166) Hostd[2098975]: --> eventTypeId = "esx.problem.vm.kill.unexpected.forcefulPageRetire.64.2",
YYYY-MM-DDT HH:MM:SS In(166) Hostd[2098975]: --> arguments = (vmodl.KeyAnyValue) [
YYYY-MM-DDT HH:MM:SS In(166) Hostd[2098975]: --> (vmodl.KeyAnyValue) {
YYYY-MM-DDT HH:MM:SS In(166) Hostd[2099324]: [Originator@6876 sub=Vimsvc.##-########] Event 29630 : ###-###-####-####-##.3.1 contained the host >physical page ######## which was scheduled for immediate retirement. To avoid system instability the virtual machine is forcefully >powered off.

     

  • Corresponding entries are visible in the /var/run/log/vmkernel.log:
    YYYY-MM-DDT HH:MM:SS Al(177) vmkalert: cpu35:########)ALERT: MCA: 191: UCNA Poll G0 B11 Sfc00cd00004000c2 A39ec78c80 M9000aa822088086 >P39ec78c80/40 Memory Controller Scrubbing Error on Channel 2.
YYYY-MM-DDT HH:MM:SS In(182) vmkernel: cpu35:########)MCAIntel: 1362: Force retiring MPN 0x##### to recover from MCA error detected by cpu35 in >bank17.
YYYY-MM-DDT HH:MM:SS In(182) vmkernel: cpu32:########)NetPort: 1887: disabled port 0x######
YYYY-MM-DDT HH:MM:SS In(182) vmkernel: cpu63:########)NetPort: 708: Failed to acquire port non-exclusive lock 0x4000018[Failure].
YYYY-MM-DDT HH:MM:SS In(182) vmkernel: cpu32:########)Net: 3834: dissociate dvPort #### from port 0x######
YYYY-MM-DDT HH:MM:SS In(182) vmkernel: cpu32:########)Net: 3841: disconnected client from port 0x######

     

  • Memory check exceptions are documented in the /var/run/log/vobd.log as follows:
    YYYY-MM-DDT HH:MM:SS In(14) vobd[2097814]: [cpuCorrelator] 5604332040761us: [vob.cpu.mce.log4] MCE bank 7: status:0x########>misc:0####02c11##01086 addr:0x######## cpu:1 physAddr:0x######## physSize:0x40 ceCount:0x1
YYYY-MM-DDT HH:MM:SS In(14) vobd[2097814]: [cpuCorrelator] 5604333223462us: [vob.cpu.mce.log4] MCE bank 7: status:0x######## >misc:0####02c11##01086 addr:0x######## cpu:23 physAddr:0x######## physSize:0x40 ceCount:0x9
YYYY-MM-DDT HH:MM:SS In(14) vobd[2097814]: [VMCorrelator] 5604334936215us: [vob.vm.kill.unexpected.forcefulPageRetire.64] The virtual machine >using the configuration file /vmfs/volumes/########-########-####-############/##-###-#####/##-###-#####.vmx contains the host physical >page 0x######## that was scheduled for immediate retirement. To avoid system instability, the virtual machine has been powered off.
YYYY-MM-DDT HH:MM:SS In(14) vobd[2097814]: [VMCorrelator] 5604914108597us: [esx.problem.vm.kill.unexpected.forcefulPageRetire.64.2] >/vmfs/volumes/########-########-####-############/##-###-#####/##-###-#####.vmx contained the host physical page 0x######## which was >scheduled for immediate retirement. To avoid system instability the virtual machine is forcefully powered off.
YYYY-MM-DDT HH:MM:SS In(14) vobd[2097814]: [cpuCorrelator] 5604341340433us: [vob.cpu.mce.log4] MCE bank 17: status:0x######## >misc:0####02c11##01086 addr:0x######## cpu:33 physAddr:0x######## physSize:0x40 ceCount:0x17c
YYYY-MM-DDT HH:MM:SS In(14) vobd[2097814]: [pageretireCorrelator] 5604348367281us: [vob.pageretire.selectedmpnthreshold.host.exceeded] Number of >MPNs selected for retirement is 4
YYYY-MM-DDT HH:MM:SS In(14) vobd[2097814]: [VMCorrelator] 5604348367305us: [vob.vm.kill.unexpected.forcefulPageRetire.64] The virtual machine >using the configuration file /vmfs/volumes/########-########-####-############/###-###-####-####/###-###-####-####.vmx contains the >host physical page 0x######## that was scheduled for immediate retirement. To avoid system instability, the virtual machine has been >powered off.
YYYY-MM-DDT HH:MM:SS In(14) vobd[2097814]: [VMCorrelator] 5604927462883us: [esx.problem.vm.kill.unexpected.forcefulPageRetire.64.2] >/vmfs/volumes/########-########-####-############/###-###-####-####/###-###-####-####.vmx contained the host physical page 0x######## >which was scheduled for immediate retirement. To avoid system instability the virtual machine is forcefully powered off.

Environment

VMware vSphere ESXi 8.0.x

VMware vSphere ESXi 7.0.x

Cause

The host experienced a recoverable Machine Check Exception (MCE) that did not escalate to a full host crash. The error was isolated to a specific memory page. As a protective measure, the Virtual Machine consuming that page was shut down to prevent data corruption. 

Resolution

To resolve this issue, coordinate with the hardware vendor to conduct the following diagnostics.

  1. Engage the Original Equipment Manufacturer (OEM) to perform a deep diagnostic of the physical memory modules.

  2. Review system hardware logs (SEL, IML, iDRAC, iLO, XCC) to correlate ESXi log timestamps with the physical DIMM slot location.

  3. Replace the identified faulty memory module (DIMM) to resolve recurring Machine Check Error (MCE) events.
Powered by Wolken Software