Time on BOSH deployed VM is out of synchronization

Products

• Operations Manager

Issue/Introduction

A virtual machines (VM) deployed by BOSH with xenial stemcell is found to be out of synchronization of time with NTP servers. Time synchronization on VMs deployed by BOSH with xenial stemcell is managed by chrony service.

There is a known issue in some xenial stemcell releases where the chrony service occasionally crashes or exits by itself and won't be restarted automatically. 

Checking the status of chrony service will show something like this:

# systemctl status chrony.service
● chrony.service - LSB: Controls chronyd NTP time daemon
   Loaded: loaded (/etc/init.d/chrony; bad; vendor preset: enabled)
   Active: active (exited) since Sun 2020-04-19 18:00:40 UTC; 1 months 16 days ago
     Docs: man:systemd-sysv-generator(8)
  Process: 11178 ExecStop=/etc/init.d/chrony stop (code=exited, status=0/SUCCESS)
  Process: 11185 ExecStart=/etc/init.d/chrony start (code=exited, status=0/SUCCESS)

Jun 05 05:42:43 7f4138ca-3123-4c80-8f9e-3c1139b357ac chronyd[11190]: chronyd exiting

This means that the chrony service had exited some time ago and was not restarted automatically.

Environment

Product Version: 2.7

Resolution

The issue was fixed in new xenial stemcell releases by adding auto-restart functionality to chrony service. 

If the following messages are shown with the systemctl status chrony.service command, it means that the auto-restart functionality is included.

  Drop-In: /etc/systemd/system/chrony.service.d
           └─restart.conf

The temporary solution is to restart the chrony service on the VM with the following:

# systemctl restart chrony.service

The permanent fix is to upgrade to latest xenial stemcell version supporting the BOSH deployment to get a permanent fix. For example, bosh-vsphere-esxi-ubuntu-xenial-go_agent/456.104+. 

 

More resources about the chrony service can be found in this article.