Timekeeping best practices for Linux guests

Products

VMware VMware Desktop Hypervisor VMware vSphere ESXi

Issue/Introduction

This article presents best practices for Linux timekeeping.

There is also a description of the recommended settings and usage for NTP time sync, configuration of VMware Tools time synchronization, and Virtual Hardware Clock configuration, to achieve best timekeeping results.

Environment

VMware ESXi
VMware vSphere

Cause

The performance of guest system timekeeping in virtual machines is subject to all of the factors that typically cause time to drift in any system.

Virtualization overheads and life cycle events introduce additional system factors that can affect timekeeping mechanisms to cause time drifts.

Resolution

Linux guest timekeeping best practices:

Use NTP

VMware recommends using NTP instead of VMware Tools periodic time synchronization. NTP is an industry standard network time synchronization program, which ensures accurate timekeeping in your guest.
It may be necessary to open the firewall (UDP 123) to allow NTP traffic.

There are various implementations of the NTP client program, including ntpd (the reference NTP Client implementation), chrony, and other commercial and open source offerings.
VMware recommends using the NTP client program recommended by the vendor of your specific Linux distribution.

In general, follow standard best practices for NTP.
NTP services need 3 or more NTP servers for optimum operation.

Reference: https://www.pool.ntp.org/join/configuration.html

To work properly ntpd needs to talk to at least 3 servers ("A man with a watch knows what time it is. A man with two watches is never sure").
For servers in the pool we recommend configuring no less than 3 and no more than 7 servers.

Choose a set of servers to synchronize to that have accurate time and adequate redundancy.
If you have many virtual or physical client machines to synchronize, set up some internal servers for them to use, so that all your clients are not directly accessing an external low-stratum NTP server and overloading it with requests.

ntpd

ntpd is a widely used implementation of Network Time Protocol. Please refer to your operating system vendor's documentation for information on configuring and using ntpd.
Additionally, following are the best practices when using ntpd in VMware virtual machines.

Allow large time jumps

Virtual machine life-cycle events, such as resume from suspend, may result in large time drifts or time jumps that cause NTP to give up. Use the following configuration directive to instruct ntpd to not give up in such cases:

tinker panic 0

Important: This configuration directive must be at the top of the configuration file (ntp.conf).

Do not use local clock as a time source

It is also important to not use the local clock as a time source, often referred to as the Undisciplined Local Clock. ntpd has a tendency to fall back to this in preference to the remote servers when there is a large amount of time drift. An example of such a configuration is:

server 127.127.1.0
fudge 127.127.1.0 stratum 10

Remove these lines (and restart ntpd) to stop this behavior.

DoS amplification attack (CVE-2013-5211)

Important: The DoS amplification attack described in CVE-2013-5211 affects versions of NTP before 4.2.7p26.

For information on how CVE-2013-5211 affects VMware products, see Mitigation and Remediation for NTP DDoS attack in ESX/ESXi and vCenter Server Appliance (CVE-2013-5211) .

You can check the version currently running on your system by running one of these commands:

ntpd --version
or,
ntpd -c rv

If you are running a version older than 4.2.7p26, add the following lines to your ntp.conf file to mitigate this attack:

restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict -6 ::1

Note: Some Linux distributions back-port security fixes such as the one described in CVE-2013-5211 without updating version information. Others show detailed information. Review the package security information regarding the distribution used for your VM. Example for Debian: https://security-tracker.debian.org/tracker/CVE-2013-5211

VMware Tools time synchronization

Disable periodic time synchronization

Only a single time synchronization program should be disciplining the time of an operating system. Therefore, when using NTP in the guest, you must ensure that VMware Tools periodic time synchronization is disabled (the default setting for VMware virtual machines).

See KB 326306 for information on how to disable periodic time synchronization.

Use one-off time synchronization

Certain virtual machine life-cycle events, such as resuming from vMotion or a snapshot, can cause guest clock to become incorrect (typically lag behind real wall clock time). VMware Tools recognizes the lag, and synchronizes guest operating system time to that of the host. This capability is turned on by default and recommended for use.

See KB 326306 for information on how to disable one-off time synchronization. (Not Recommended)

Important: Since one-off time synchronization relies on the time in the host operating system as a reference, it is important that host system time is kept accurate using time synchronization software (such as NTP) according to best practices for that host.

See KB 318545 for ESX and ESXi time keeping best practices.

Virtual hardware clock configuration

When configuring the Linux guest operating system, if you are given a choice between keeping the "hardware" clock (that is, the virtual CMOS time of day clock) in UTC or local time, choose UTC. This avoids any confusion when your local time changes between standard and daylight savings time (or summer time in some countries).

For more information, see Timekeeping in VMware Virtual Machines.

Linux operating system distribution and versions

For best time keeping performance, use the latest stable versions of supported Linux guest operating systems. See the guest OS compatibility list for Linux operating system distributions and the specific versions, supported by VMware.