Virtual Machine powered off with error: <vm name> contained the host physical page ### which was scheduled for immediate retirement.
search cancel

Virtual Machine powered off with error: <vm name> contained the host physical page ### which was scheduled for immediate retirement.

book

Article ID: 404366

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  •  Virtual Machines shut down unexpectedly.

  •  Events display messages like:

Description: <vm name> contained the host physical page ### which was scheduled for immediate retirement. To avoid system instability the virtual machine is forcefully powered off.

Event ID: esx.problem.vm.kill.unexpected.forcefulPageRetire.64.2

  • Comparable entries can be found within the host daemon log at /var/run/log/hostd.log:
    [timestamp] In(166) Hostd[2099322]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/########-########-####-############/##-###-#####/##->###-#####.vmx] Deferring power on completion until VM is at a stable state
[timestamp] In(166) Hostd[2098975]: --> eventTypeId = "esx.problem.vm.kill.unexpected.forcefulPageRetire.64.2",
[timestamp] In(166) Hostd[2098975]: --> arguments = (vmodl.KeyAnyValue) [
[timestamp] In(166) Hostd[2098975]: --> (vmodl.KeyAnyValue) {
[timestamp] In(166) Hostd[2099324]: [Originator@6876 sub=Vimsvc.##-########] Event 29630 : ###-###-####-####-##.3.1 contained the host >physical page ######## which was scheduled for immediate retirement. To avoid system instability the virtual machine is forcefully >powered off.

     

  • Corresponding entries are visible in the /var/run/log/vmkernel.log:
    [timestamp] Al(177) vmkalert: cpu35:########)ALERT: MCA: 191: UCNA Poll G0 B11 Sfc00cd00004000c2 A39ec78c80 M9000aa822088086 >P39ec78c80/40 Memory Controller Scrubbing Error on Channel 2.
[timestamp] In(182) vmkernel: cpu35:########)MCAIntel: 1362: Force retiring MPN 0x##### to recover from MCA error detected by cpu35 in >bank17.
[timestamp] In(182) vmkernel: cpu32:########)NetPort: 1887: disabled port 0x######
[timestamp] In(182) vmkernel: cpu63:########)NetPort: 708: Failed to acquire port non-exclusive lock 0x4000018[Failure].
[timestamp] In(182) vmkernel: cpu32:########)Net: 3834: dissociate dvPort #### from port 0x######
[timestamp] In(182) vmkernel: cpu32:########)Net: 3841: disconnected client from port 0x######

     

  • Memory check exceptions are documented in the /var/run/log/vobd.log as follows:
    [timestamp] In(14) vobd[2097814]: [cpuCorrelator] 5604332040761us: [vob.cpu.mce.log4] MCE bank 7: status:0x########>misc:0x200802c110801086 addr:0x######## cpu:1 physAddr:0x######## physSize:0x40 ceCount:0x1
[timestamp] In(14) vobd[2097814]: [cpuCorrelator] 5604333223462us: [vob.cpu.mce.log4] MCE bank 7: status:0x######## >misc:0x200805c2b0001086 addr:0x######## cpu:23 physAddr:0x######## physSize:0x40 ceCount:0x9
[timestamp] In(14) vobd[2097814]: [VMCorrelator] 5604334936215us: [vob.vm.kill.unexpected.forcefulPageRetire.64] The virtual machine >using the configuration file /vmfs/volumes/########-########-####-############/##-###-#####/##-###-#####.vmx contains the host physical >page 0x######## that was scheduled for immediate retirement. To avoid system instability, the virtual machine has been powered off.
[timestamp] In(14) vobd[2097814]: [VMCorrelator] 5604914108597us: [esx.problem.vm.kill.unexpected.forcefulPageRetire.64.2] >/vmfs/volumes/########-########-####-############/##-###-#####/##-###-#####.vmx contained the host physical page 0x######## which was >scheduled for immediate retirement. To avoid system instability the virtual machine is forcefully powered off.
[timestamp] In(14) vobd[2097814]: [cpuCorrelator] 5604341340433us: [vob.cpu.mce.log4] MCE bank 17: status:0x######## >misc:0x900222208088086 addr:0x######## cpu:33 physAddr:0x######## physSize:0x40 ceCount:0x17c
[timestamp] In(14) vobd[2097814]: [pageretireCorrelator] 5604348367281us: [vob.pageretire.selectedmpnthreshold.host.exceeded] Number of >MPNs selected for retirement is 4
[timestamp] In(14) vobd[2097814]: [VMCorrelator] 5604348367305us: [vob.vm.kill.unexpected.forcefulPageRetire.64] The virtual machine >using the configuration file /vmfs/volumes/########-########-####-############/###-###-####-####/###-###-####-####.vmx contains the >host physical page 0x######## that was scheduled for immediate retirement. To avoid system instability, the virtual machine has been >powered off.
[timestamp] In(14) vobd[2097814]: [VMCorrelator] 5604927462883us: [esx.problem.vm.kill.unexpected.forcefulPageRetire.64.2] >/vmfs/volumes/########-########-####-############/###-###-####-####/###-###-####-####.vmx contained the host physical page 0x######## >which was scheduled for immediate retirement. To avoid system instability the virtual machine is forcefully powered off.

Environment

VMware vSphere ESXi 8.0.x

VMware vSphere ESXi 7.0.x

Cause

The host experienced a recoverable Machine Check Exception (MCE) that did not escalate to a full host crash. The error was isolated to a specific memory page. As a protective measure, the Virtual Machine consuming that page was shut down to prevent data corruption. 

Resolution

To resolve this issue, coordinate with the hardware vendor to conduct the following diagnostics.

  1. Engage the Original Equipment Manufacturer (OEM) to perform a deep diagnostic of the physical memory modules.
  2. Review system hardware logs (SEL, IML, iDRAC, iLO, XCC) to correlate ESXi log timestamps with the physical DIMM slot location.
  3. Replace the identified faulty memory module (DIMM) to resolve recurring Machine Check Error (MCE) events.
Powered by Wolken Software