Impact of virtual machine memory and CPU resource limits

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides information about the impact of virtual machine resource limits on CPU and memory. A virtual machine experiences poor performance even though the ESXi host is not fully utilized. Ballooning and vmkernel swapping occur.

Environment

VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x

Resolution

Resource limits on a virtual machine for both CPU and memory form an artificial boundary for the virtual machine.

Memory Limits

When a memory limit is set less than the virtual machine's provisioned memory, it is considered as the upper boundary for the amount of physical memory that can be directly assigned to this particular virtual machine. The guest operating system is not aware of this limit, and it optimizes memory management options to the assigned memory size.

 

When the limit is reached or exceeded, the guest operating system can still request new pages, but due to the limit the VMkernel does not allow the guest to directly consume more physical memory and treats the virtual machine as if the resource is under contention. Memory reclamation techniques are used to enable the virtual machine to consume requested memory.

 

Depending on the number of pages requested by the virtual machine, the VMkernel might resort to VMkernel swap to fulfil the request, in the worst case scenario. The VMkernel first tries to reclaim memory by inflating the Balloon Driver to let the guest memory manager decide what to page out. The VMkernel also tries to compress memory pages before swapping them out.

 

You can verify the impact of a memory limit by running esxtop and looking at MCTLSZ and MCTLTGT, SWCUR and SWTGT, and CACHEUSD.

 

For more information on using esxtop, see Memory Panel statistics.

 

CPU Limits

 

When a CPU Limit is set on a virtual machine resource setting, the virtual machine is deliberately held from being scheduled to a PCPU when it has used up its allocated CPU resource. This happens regardless of the CPU utilization. The specified limit is applied to entire virtual machine CPU resources, not to guest OS. If the limit is set to 500MHz, the virtual machine is rescheduled from the PCPU and has to wait before it is allowed to be scheduled again. As such, the virtual machine might experience performance degradation. If many virtual machines have the limit on the same ESXi host, they might impact on entire ESXi activities.

 

Note: For an SMP virtual machine, the sum of all vCPUs cannot exceed the specified limit. For example, 4 vCPU virtual machine with a limit of 1200MHz and equal load among vCPUs would result in a max of 300MHz per vCPU.

 

You can verify the impact of a CPU limit by running esxtop and looking at %RDY and %MLMTD.

 

For more information on using esxtop, see CPU Panel statistics.

Additional Information

You can also use shares to restrict resource usage of certain virtual machines only when the host becomes overcommitted.

 

You can use a limit to simulate resource over commitment in advance or to dynamically constrain the resource usage of a virtual machine (for example, in case a rogue virtual machine's excessive resource utilization negatively affects the host and the other virtual machines).

For more information on how memory and CPU management work:

• In ESXi 7.0, see the vSphere Resource Management 7.0 Guide.
• In ESXi 8.0, see the vSphere Resource Management 8.0 Guide