VM Power ON or vMotion operations fails with error "Could not power on virtual machine: Admission check failed for memory resource"
search cancel

VM Power ON or vMotion operations fails with error "Could not power on virtual machine: Admission check failed for memory resource"

book

Article ID: 301491

calendar_today

Updated On:

Products

VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

  • When DRS decides to Power ON or vMotion the VM, the target host doesn't have sufficient resources to allow the operation from proceeding.

  • VM powers ON works fine when DRS is set to manual.

  • vCenter Server log (/var/log/vmware/vpxd/vpxd.log) shows below error messages:

    /var/log/vmware/vpxd/vpxd.log

    <Date & Time> info vpxd[07602] [Originator@6876 sub=Default opID=lro-181744072-5632bbd1-01] [VpxLRO] -- ERROR task-987370 -- <VM_Name> -- Drm.ExecuteVMotionLRO: vmodl.fault.SystemError:
    --> Result:
    --> (vmodl.fault.SystemError) {
    -->    faultCause = (vim.fault.GenericVmConfigFault) {
    -->       faultCause = (vmodl.MethodFault) null, 
    -->       faultMessage = (vmodl.LocalizableMessage) [
    -->          (vmodl.LocalizableMessage) {
    -->             key = "msg.moduletable.powerOnFailed", 
    -->             arg = (vmodl.KeyAnyValue) [
    -->                (vmodl.KeyAnyValue) {
    -->                   key = "1", 
    -->                   value = "MonitorLoop"
    -->                }
    -->             ], 
    -->             message = "Module 'MonitorLoop' power on failed. "
    -->          }, 
    -->          (vmodl.LocalizableMessage) {
    -->             key = "msg.migrate.resume.fail", 
    -->             arg = <unset>, 
    -->             message = "The VM failed to resume on the destination during early power on. "
    -->          }, 
    -->    ], 
    -->    reason = "Launch failure"
     
  • Virtual Machine logs under respective VM storage location shows below errors :

    vmware.log

    <Date & Time> In(05) vmx - Msg_Post: Error
    <Date & Time> In(05) vmx - [vob.sched.group.mem.invalid3] Group vmm.315706559: Failed to apply memory configuration for VMM group vmm0:VM_Name. (min: 33554432, max: 33554432, minLimit: 18446744073709551615, shares: 18446744073709551613, units: pages)
    <Date & Time> In(05) vmx - [msg.vmmonVMK.admitFailed] Could not power on virtual machine: Admission check failed for memory resource.
    <Date & Time> In(05)+ vmx - See the VMware ESX Resource Management Guide for information on resource management settings.
    <Date & Time> In(05) vmx - [msg.monitorLoop.createVMFailed.vmk] Failed to power on VM.
    2024-01-17T19:27:04.243Z In(05) vmx - ----------------------------------------
  • hostd.log on destination host (during vMotion) shows below errors:

    /var/run/logs/hostd.log

    <Date & Time> verbose hostd[255724550] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/<Datastore_UUID>/<VM_Name.vmx> opID=<opId>] Handling vmx message 234488962: Group vmm.315706559: Failed to apply memory configuration for VMM group vmm0:VM_Name. (min: 33554432, max: 33554432, minLimit: 18446744073709551615, shares: 18446744073709551613, units: pages)
    --> Could not power on virtual machine: Admission check failed for memory resource.
    --> See the VMware ESX Resource Management Guide for information on resource management settings.
    --> Failed to power on VM.

Cause

This issue is because of an internal inconsistency regarding memory reservation between VC/DRS and ESXi/Kernel. This happens when reducing the reservation from a monster VM significantly, where the VM reservation reduction succeeds in the DRS but fails on the host kernel.

Resolution

This issue is resolved in vCenter Server 8.0 U3.


To workaround the issue:

To fix the inconsistency of VM, try any one of the options below on the virtual machine where memory reservation was changed from large size to 0 MB:

Option 1 :

  • Powering off the VM (on which memory reservation was changed from large size to 0 MB) and then power on the VM.

Option 2 (try this option if Option 1 doesn't help):

  • Shutdown the VM
  • Set sched.mem.min = "0", sched.mem.minSize = "0" to the VMX file of VM.
  • Power On the VM

Option 3 :

  • Go to VM setting in the vCenter UI and configure the VM with the same reservation as before. 
  • Wait one minute then decrease the reservation by small decrements, until the reservation is back to 0. Allow one minute between each decrement of the reservation.
  • SSH to the ESXi host and get VMX cartel ID using the command "esxcli vm process list".
  • Verify the change has taking effect on the host by checking the values, particularly "min", from the following command. Use the cartel ID from above step to identify the relevant VM ID from the output of memstats:

    memstats -r vm-stats -u MB -s name:memSize:min:max:minLimit