Large VM power on sometimes failed

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
This issue found in VMware vSphere ESXi 6.5.

Large VM power on sometimes failed with the following errors.

In the vSphere Client, you see entries similar to:

An error was received from the ESX host while powering on VM .
 - Failed to start the virtual machine. Module ‘Disk  power on failed.
 - File system specific implementation of LookupAndOpen[file] failed
 - Remote disconnected Failed to establish transport connection (9): There is no VMware process running for config file "datastore/directory/VMNAME.vmx "

In the vmware.log, you see entries similar to:
- Could not power on virtual machine: Out of memory.
- PANIC: VERIFY bora/mks/main/mksThread.c:96

Environment

VMware vSphere ESXi 6.5

Cause

Automatically assigned Overhead memory of Large VM is too tight for power on.

Resolution

Fixed in vSphere 7

Workaround:
***Before you execute this workaround please open a support request.****
Before the following steps you need to try a VM power on several times for a VM successfully powered on. 


Increase Configured memory reservation.


1. confirm VMX Cartel ID of a Virtual Machin with the following command
esxcli vm process list

example result)
test256VM
   World ID: 191763
   Process ID: 0
   VMX Cartel ID: 191762 ****
   UUID: 56 4d b1 61 4c 53 ## ##-## ## ## d5 30
   Display Name: test256VM
   Config File: /vmfs/volumes/########-####-########d470/test256VM/test256VM.vmx

VMX Cartel ID is 191762 in this sample

2. confirm peak value for requested memory reservation with the following command
   memstats -r group-stats -s rMinPeak:name:min -u mb


example result)
In this example you seek uw.191762

-----------------------------------------------------------
     gid                         name        min   rMinPeak
-----------------------------------------------------------
...
  514242                    uw.191762         39         73*****

min is Configured memory reservation.
rMinPeak is peak value for requested memory reservation.

In this example 73 is peak value for requested memory reservation.

3. set Configured memory reservation
   1) Edit setting>VM options>Advanced>Edit Configuration
   2) Add parameter sched.mem.vmxMin
   3) set value greater than rMinPeak value.
   
We need change min greater than rMinPeak value. In this example min should be greater than 73.
So you need to set sched.mem.vmxMin rMinPeak value + 10% buffer MB.