Time drift noticed after vMotion in a VM using ACPI PM timer based clock source

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Time drifts in a VM after it is migrated to a host running ESXi version 7.0, 7.0 U1 or 7.0 U2.
• Guest time sync agents, such as ntpd, report loss of time synchronization.
• CPU speed of the source and destination hosts are different.
• The guest OS is using ACPI PM timer as its primary clock source.

Below is an example for checking clock source in a Linux virtual machine
[root@VM ~]# cat /sys/devices/system/clocksource/clocksource0/current_clocksource acpi_pm

Below is an example for checking clock source in a FreeBSD-based virtual machine
[root@VM ~]sysctl kern.timecounter.hardware
kern.timecounter.hardware: TSC-low

For more information, refer to your specific guest OS vendor's documentation.

Environment

VMware vSphere ESXi 7.0.2
VMware vSphere ESXi 7.0.3
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 7.0.1

Cause

A bug in the emulation of the virtual ACPI PM timer device causes it to tick faster (or slower) after the virtual machine is migrated to a CPU with a significantly different frequency. This bug is present in ESXi 7.0, 7.0 U1, and 7.0 U2, and has been fixed in ESXi version 7.0.3.

Resolution

The issue is fixed in version ESXi 7.0 U3. For more information, see VMware ESXi 7.0 Update 3c Release Notes .

Note: A small amount of clock drift, in the order of tens or hundreds of milliseconds, is expected after vMotion due to VM stun. ntpd is expected to correct this time lag, and time synchronization should stabilize eventually. For more information, see Migrating virtual machine may cause guest operating system clock to fall behind real time (2108828)

Workaround:

To workaround this issue follow any of the below workarounds.

Update the kernel to not use ACPI PM timer as a clock source.

Ensure to follow best practices for Guest OS distribution as well as VMware timekeeping best practices where applicable. Remove any clock source overrides if present.
For e.g. on Linux, remove any boot parameters like "clock=pmtmr divider=4" from the kernel command-line.

After following best practices, if kernel still uses ACPI PM clock source, you will need to override it.

Overriding clock source in a Linux VM:

List available clock sources with:
[root@VM ~]# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
tsc hpet acpi_pm

You can select 'tsc' as the clock source:
[root@VM ~]# echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource

Overriding clock source in a FreeBSD VM:

List available clock sources with:
[root@VM ~]# sysctl kern.timecounter.choice
kern.timecounter.choice: ACPI-fast(900) i8254(0) HPET(950) TSC-low(1000) dummy(-1000000)

You can select 'tsc' as the clock source:
[root@VM ~]# sysctl kern.timecounter.hardware=TSC-low

However, this clock source selection is not persistent across reboots. To make it persistent, add "kern.timecounter.hardware=TSC-low" to /etc/sysctl.conf and reboot.

Affinitize Virtual Machine to a host group:

You can also work around this problem by making sure this Virtual Machine is migrated only to a host with matching CPU frequency as the source host, by creating a VM-host affinity rule. For more information see vSphere Resource Management Guide .

Additional Information

Timekeeping best practices for Linux guests

Timekeeping best practices for Windows