Timekeeping best practices for Linux guests

Products

VMware VMware Desktop Hypervisor VMware vSphere ESXi

Issue/Introduction

This article presents best practices for Linux timekeeping. These recommendations include specifics on the particular kernel command line options to use for the Linux operating system of interest. There is also a description of the recommended settings and usage for NTP time sync, configuration of VMware Tools time synchronization, and Virtual Hardware Clock configuration, to achieve best timekeeping results.

Environment

VMware ESX 4.0.x
VMware ESXi 4.1.x Embedded
VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.5
VMware ESXi 4.0.x Installable
VMware vSphere ESXi 6.0
VMware ESXi 4.1.x Installable
VMware ESX Server 3.5.x
VMware vSphere ESXi 5.1
VMware ESX 4.1.x
VMware ESXi 4.0.x Embedded

Cause

The performance of guest system timekeeping in virtual machines is subject to all of the factors that typically cause time to drift in any system. Virtualization overheads and life cycle events introduce additional system factors that can affect timekeeping mechanisms to cause time drifts.

Resolution

Linux guest timekeeping best practices:

Use NTP

VMware recommends using NTP instead of VMware Tools periodic time synchronization. NTP is an industry standard network time synchronization program, which ensures accurate timekeeping in your guest. It may be necessary to open the firewall (UDP 123) to allow NTP traffic.

There are various implementations of the NTP client program, including ntpd (the reference NTP Client implementation), chrony, and other commercial and open source offerings. VMware recommends using the NTP client program recommended by the vendor of your specific Linux distribution. In general, follow standard best practices for NTP. Choose a set of servers to synchronize to that have accurate time and adequate redundancy. If you have many virtual or physical client machines to synchronize, set up some internal servers for them to use, so that all your clients are not directly accessing an external low-stratum NTP server and overloading it with requests.

ntpd

ntpd is a widely used implementation of Network Time Protocol. Please refer to your operating system vendor's documentation for information on configuring and using ntpd. Additionally, following are the best practices when using ntpd in VMware virtual machines.

Allow large time jumps

Virtual machine life-cycle events, such as resume from suspend, may result in large time drifts or time jumps that cause NTP to give up. Use the following configuration directive to instruct ntpd to not give up in such cases:

tinker panic 0

Important: This configuration directive must be at the top of the configuration file (ntp.conf).

Do not use local clock as a time source

It is also important to not use the local clock as a time source, often referred to as the Undisciplined Local Clock. ntpd has a tendency to fall back to this in preference to the remote servers when there is a large amount of time drift. An example of such a configuration is:

server 127.127.1.0
fudge 127.127.1.0 stratum 10

Remove these lines (and restart ntpd) to stop this behavior.

DoS amplification attack (CVE-2013-5211)

Important: The DoS amplification attack described in CVE-2013-5211 affects versions of NTP before 4.2.7p26.

For information on how CVE-2013-5211 affects VMware products, see Mitigation and Remediation for NTP DDoS attack in ESX/ESXi and vCenter Server Appliance (CVE-2013-5211) .

You can check the version currently running on your system by running one of these commands:

ntpd --version
or,
ntpd -c rv

If you are running a version older than 4.2.7p26, add the following lines to your ntp.conf file to mitigate this attack:

restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict -6 ::1

Note: Some Linux distributions back-port security fixes such as the one described in CVE-2013-5211 without updating version information. Others show detailed information. Review the package security information regarding the distribution used for your VM. Example for Debian: https://security-tracker.debian.org/tracker/CVE-2013-5211

VMware Tools time synchronization

Disable periodic time synchronization

Only a single time synchronization program should be disciplining the time of an operating system. Therefore, when using NTP in the guest, you must ensure that VMware Tools periodic time synchronization is disabled (the default setting for VMware virtual machines).

See KB 326306 for information on how to disable periodic time synchronization.

Use one-off time synchronization

Certain virtual machine life-cycle events, such as resuming from vMotion or a snapshot, can cause guest clock to become incorrect (typically lag behind real wall clock time). VMware Tools recognizes the lag, and synchronizes guest operating system time to that of the host. This capability is turned on by default and recommended for use.

See KB 326306 for information on how to disable one-off time synchronization. (Not Recommended)

Important: Since one-off time synchronization relies on the time in the host operating system as a reference, it is important that host system time is kept accurate using time synchronization software (such as NTP) according to best practices for that host.

See KB 318545 for ESX and ESXi time keeping best practices.

Virtual hardware clock configuration

When configuring the Linux guest operating system, if you are given a choice between keeping the "hardware" clock (that is, the virtual CMOS time of day clock) in UTC or local time, choose UTC. This avoids any confusion when your local time changes between standard and daylight savings time (or summer time in some countries).

For more information, see Timekeeping in VMware Virtual Machines.

Linux operating system distribution and versions

For best time keeping performance, use the latest stable versions of supported Linux guest operating systems. See the guest OS compatibility list for Linux operating system distributions and the specific versions, supported by VMware.

Kernel parameters for accurate time keeping in older guests

Certain older flavors of Linux distributions require additional kernel parameters to ensure time keeping accuracy, summarized in the table below. (These kernel parameters need to be edited into the boot loader configuration, adding to the kernel parameters already configured by the distribution. See your Linux distribution's documentation for more information on how to do that.)

Most guests in the below table are either unsupported by VMware or EOL'ed by the vendor. The table serves as a historical reference only.

64-bit guests

Linux Distribution and Version	Kernel Parameters	Notes
CentOS 5.4 through 5.8		No additional kernel parameters required. Optional use of divider=10 is recommended for improved timekeeping accuracy.
CentOS 5.2 and 5.3	notsc divider=10 nohpet
CentOS 5.1	notsc nohpet
Debian 4.x	notsc nohpet
OEL 5.4 through 5.8		No additional kernel parameters required. Optional use of divider=10 is recommended for improved timekeeping accuracy.
OEL 5.2 and 5.3	notsc divider=10 nohpet
OEL 5.1	notest nohpet
RHEL 5.4 through 5.8		No additional kernel parameters required. Optional use of divider=10 is recommended for improved timekeeping accuracy.
RHEL 5.2 and 5.3 64-bit	notsc divider=10 nohpet
RHEL 5.1 with RHSA-2007:0993-13	notsc divider=10 nohpet
RHEL 5.1 without RHSA-2007:0993-13	notsc nohpet
SLED 10	clock=pmtmr nohpet
SLES 10 and SLES 10 SP1	notsc nohpet
Suse Linux 10.3	clocksource=acpi_pm
Suse Linux 10 through 10.2	notsc nohpet
Ubuntu 9.04		Avoid using kernel 2.6.28-7.18, known to cause guest operating system to stop responding.
Ubuntu 8.04, 8.10	clocksource=acpi_pm	Avoid using kernels 2.6.24-24.52 and 2.6.28-12.28, known to cause guest operating system to stop responding.

32-bit guests

Linux Distribution and Version	Kernel Parameters	Notes
CentOS 5.4 through 5.8		No additional kernel parameters required. Optional use of divider=10 is recommended for improved timekeeping accuracy.
CentOS 5.1 through 5.3	clocksource=acpi_pm divider=10
CentOS 5.0	clocksource=acpi_pm
Debian 4.x	clocksource=acpi_pm divider=10
OEL 5.4 through 5.8		No additional kernel parameters required. Optional use of divider=10 is recommended for improved timekeeping accuracy.
OEL 5.1 through 5.3	clocksource=acpi_pm divider=10
OEL 5.0	clocksource=acpi_pm
RHEL 5.4 through 5.8		No additional kernel parameters required. Optional use of divider=10 is recommended for improved timekeeping accuracy.
RHEL 5.1 through 5.3	clocksource=acpi_pm divider=10
RHEL 5.0	clocksource=acpi_pm
SLES 10 SP1, SP2, SP3, SP4	clock=pmtmr hpet=disable
Suse Linux 10.2 and 10.3	clocksource=acpi_pm
Suse Linux 10, 10.1	clock=pmtmr hpet=disable
Ubuntu 9.04		Avoid using kernel prior to 2.6.28-7.18 if possible. May cause guest operating system to stop responding. EOL
Ubuntu 8.10	clocksource=acpi_pm	Avoid using kernel prior to 2.6.27-12.28 if possible. May cause guest operating system to stop responding. EOL

Notes on "divider=10" usage

For some operating systems, divider=10 is a supported kernel configuration option, but might not be necessary for accurate timekeeping. Using it reduces the frequency of timer interrupts by 10x, which reduces the CPU overhead of processing timer interrupts. This overhead is especially noticeable for idle virtual machines. The only drawback of using divider=10 is that the granularity of wake-ups provided by the kernel changes from 1 ms to 10 ms. The vast majority of applications are not affected by this, but using divider=10 may not be the right trade-off for some time-sensitive applications.

For some operating systems, specifically older versions, divider=10 greatly improves timekeeping accuracy and is strongly recommended.