Migrating virtual machine may cause guest operating system clock to fall behind real time

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides information on the clock delay on a migrated virtual machine that is configured to use NTP services for time synchronization.

Cause

When a virtual machine is migrated, its time keeping apparatus is temporarily frozen, as it is suspended, copied and resumed at a destination host.

This causes the guest operating system clock to fall behind real time, often in the order of tens or even hundreds of milliseconds.

Resolution

To resolve this issue, use one of these options:

• Run this server configuration command which uses iburst option in the guest's NTP configuration(ntp.conf) file to speed up the NTP algorithms:

server ntp-1.example.com iburst

Note: This sends a burst of NTP messages, grooming the data at a much faster pace to reduces the initial delay in setting the clock.

• Run this server configuration command which uses minpoll and maxpoll options to decrease the poll interval of NTP messages to hasten the slew adjustments to the system clock:
  
  server ntp-1.example.com minpoll 4 maxpoll 4
  
  Note: Although usage of these options with shared NTP infrastructure(such as NTP servers on the internet) is considered as poor practice, they are useful if you have dedicated internal time servers. For more information on these options and their impact on overall NTP behavior, see the NTP manual.
  
  Note: The preceding link is correct as of September 4, 2015. If you find the link is broken, provide a feedback and a VMware employee will update the link.
   
• If adjustments to NTP prove to be insufficient in mitigating effects of time differences due to virtual machine migration, configure VMware tools one-time time synchronization to have a lower threshold value.
  
  Run the command which uses vmx option pref.timeLagInMilliseconds, and defaults the time to 1000 (for 1 second):
  
  For example, if you want the guest clock to be synchronized with the host, whenever time falls behind more than 100 milliseconds after migration, add this to your vmx file.For information on editing vmx file, refer Tips for editing a .vmx file (1714).
  
  pref.timeLagInMilliseconds = 100

Additional Information

If you configure the guest to use NTP, such time difference is corrected automatically. For time differences of up to 128 milliseconds (stepthreshold) NTP corrects the difference in a continuous manner, slewing at 1 second intervals. Otherwise, it steps the clock forward to the consensus time, but only after it establishes that the clock has consistently lagged behind for over a period of 900 seconds (called the stepout threshold).

Although unlikely, if the offset happens to be greater than 1 second, VMware tools running in the guest steps the guest clock forward immediately upon resume, to match the underlying ESXi host's clock. This is known as a one-time time synchronization, performed every time a virtual machine is resumed from a checkpoint, and only if the guest clock lags behind by more than a threshold value that defaults to 1000 milliseconds. The synchronization assumes correctness of ESXi host's clock, which, as per VMware's Timekeeping best practices, should be synchronized using NTP.

These time correction mechanisms are sufficient for most guest applications. However, some applications (such as those distributed over a network) may be highly sensitive to sudden time differences, and the ntp clock discipline algorithms too slow to mitigate its effects. For instance, in its default configuration, NTP may take up to 200 seconds to slew-correct a 100 milliseconds time difference.