Intermittent Power-On Failures Observed in Large Virtual Machines
search cancel

Intermittent Power-On Failures Observed in Large Virtual Machines

book

Article ID: 327774

calendar_today

Updated On: 01-08-2025

Products

VMware vSphere ESXi VMware vSphere ESXi 7.0 VMware vSphere ESXi 6.0

Issue/Introduction

Symptoms:
This issue found in VMware vSphere ESXi 6.5.

Large VM power on sometimes failed with the following errors.

In the vSphere Client, you see entries similar to:

An error was received from the ESX host while powering on VM .
 - Failed to start the virtual machine. Module ‘Disk  power on failed.
 - File system specific implementation of LookupAndOpen[file] failed
 - Remote disconnected Failed to establish transport connection (9): There is no VMware process running for config file "datastore/directory/VMNAME.vmx "

In the vmware.log, we see entries similar to:
- Could not power on virtual machine: Out of memory.
- PANIC: VERIFY bora/mks/main/mksThread.c:96

Environment

VMware vSphere ESXi 6.5

Cause

Insufficient Overhead Memory Allocation for Powering On Large VMs

Resolution


This is Fixed in vSphere 7.0

Workaround:
***Before you execute this workaround please open a support request.****
Before the following steps you need to try a VM power on several times for a VM successfully powered on. 


Increase Configured memory reservation.


1. Confirm VMX Cartel ID of a Virtual Machin with the following command
      esxcli vm process list

   example result)
   test256VM
   World ID: 191763
   Process ID: 0
   VMX Cartel ID: 191762 ****
   UUID: 56 4d b1 61 4c 53 ## ##-## ## ## d5 30
   Display Name: test256VM
   Config File: /vmfs/volumes/########-####-########d470/test256VM/test256VM.vmx

   VMX Cartel ID is 191762 in this sample

2.  Confirm peak value for requested memory reservation with the following command
     memstats -r group-stats -s rMinPeak:name:min -u mb


    example result 
    In this example you seek uw.191762
 
    -----------------------------------------------------------
    gid                         name        min   rMinPeak
    -----------------------------------------------------------
    ...
    514242                    uw.191762         39         73*****
 
   min is Configured memory reservation.
   rMinPeak is peak value for requested memory reservation.

   In this example 73 is peak value for requested memory reservation.

3. Set Configured memory reservation
      1) Edit setting>VM options>Advanced>Edit Configuration
   2) Add parameter sched.mem.vmxMin
   3) set value greater than rMinPeak value.
   
We need change min greater than rMinPeak value. In this example min should be greater than 73.
So you need to set sched.mem.vmxMin rMinPeak value + 10% buffer MB.