Time falls behind in a virtual machine when the memory of the virtual machine is paged from disk by the VMkernel

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Time falls behind in a virtual machine
• Time on the virtual machine catches up to real time after falling behind
• Time in a virtual machine runs slower than real time

Resolution

Determining if VMkernel has swapped out memory from the virtual machine

Either esxtop or VMware Infrastructure Client can be used to gather information about VMkernel swapping activity.

To use the VI Client to determine whether the VMkernel has swapped out any of the memory from the virtual machine:

1. Click the Performance tab for the virtual machine.
2. Switch to the Memory view.
3. Add the Memory Swapped (Average) statistic using the Change Chart Options dialog.
   
   If the value is zero, the VMkernel has not swapped out any of the virtual machine memory, and this article does not apply.
   
   If the statistic is non-zero, check the Memory Swap In (Average). This statistic records the total number of pages that have been swapped in by the VMkernel for this virtual machine. If increases in this statistic are correlated with time in the virtual machine falling behind, you can attribute the falling behind to the virtual machine memory being paged.

See Timekeeping in VMware Virtual Machines for more background information on timer interrupts and timekeeping.

Can affect all kernels. Kernels with higher timer interrupt frequencies are more sensitive because as the period between timer interrupt shrinks, it gets shorter than the time required to read a page from disk. For example, with a timer interrupt frequency of 1000Hz, the period is 1ms, which is significantly shorter than a disk I/O which is approximately 4-10ms.

Solution

Run the Memory Balloon in the guest to discourage the ESX host from paging memory to disk.

Alternately, set the memory reservation equal to the memory size of the virtual machine, to reserve all of the physical memory for a virtual machine and prevent it from being paged to disk.

Other steps that can be taken to lessen the severity of the problem caused by swapping are, switching to a kernel with a lower timer rate and/or use a UP virtual machine instead of an SMP virtual machine. For more information on the timer interrupt rate of common kernels, see Determining the rate of timer interrupts a guest operating system requests (1005802) .

Note: When using the Memory Balloon driver, some memory may be paged out by the VMkernel while the virtual machine is booting before the Memory Balloon driver is loaded or there may be more memory pressure than the Memory Balloon driver can respond to.

Additional Information

If there is memory pressure on the ESX host, the VMkernel pursues several strategies for freeing up memory. It looks for pages that can be shared within and between virtual machines. It uses the memory balloon driver (vmmemctl) to try to reclaim pages from virtual machines with the guest operating system's co-operation. If these two methods don't free up enough memory, ESX may swap out pages of a virtual machine's memory to disk.

When the virtual machine references the page and the VMkernel needs to read the page from disk, it is forced to stop running the virtual machine while it waits for the disk. This can cause delivery of timer interrupts to the virtual machine to be delayed. If many pages need to be swapped in from disk in a short time period, the number of late timer interrupts can accumulate and results in time falling noticeably behind in the virtual machine.