NTP service shows as inactive on TKG Cluster in Photon OS when running timedatectl

Products

• VMware

Issue/Introduction

Symptoms:
In Photon OS, the NTP service will be shown as inactive when you run the timedatectl command as below:

capv@workload-test-md-0-7f58fcbbf4-splb4 [ ~ ]$ timedatectl
               Local time: Mon 2023-02-20 05:02:26 UTC
           Universal time: Mon 2023-02-20 05:02:26 UTC
                 RTC time: Mon 2023-02-20 05:02:26
                Time zone: UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: inactive
          RTC in local TZ: no

Environment

VMware Tanzu Kubernetes Grid 1.x

Cause

Photon OS 3.0 by default supports systemd-timesyncd service for network time synchronization and when chronyd service is used for timesync "timedatectl" commands doesn't work.

Product team identified this and provided the fix to enable Chronyd service as well for NTP along with systemd-timesyncd 

Resolution

This will be fixed in the future version of Photon OS and corresponding fix would be incorporated in TKG 

Workaround:
Please follow the below procedure to fix the NTP server issue shown as Inactive in the Photon OS with timedatectl command.

1. Set the context to the Management cluster

kubectl config set-context <management cluster context name>

2. Start patching the control plane nodes

kubectl patch -n <namespace-where-cluster-resides>  KubeadmControlPlane <kubeadm-control-plane-object> --type='json' -p='[
{"op": "add", "path": "/spec/kubeadmConfigSpec/preKubeadmCommands/-", "value": "echo \"Environment=SYSTEMD_TIMEDATED_NTP_SERVICES=chronyd.service:systemd-timesyncd.service \" >> /lib/systemd/system/systemd-timedated.service"},
{"op": "add", "path": "/spec/kubeadmConfigSpec/preKubeadmCommands/-", "value": "systemctl daemon-reload"},
{"op": "add", "path": "/spec/kubeadmConfigSpec/preKubeadmCommands/-", "value": "systemctl restart chronyd"}]'

This will roll out the Control plane nodes automatically and wait for the control plane nodes to be recreated

2. Start patching the worker nodes

kubectl  patch -n <namespace-where-cluster-resides> KubeadmConfigTemplate <kubeadm-config-template-object>  --type='json' -p='[
{"op": "add", "path": "/spec/template/spec/preKubeadmCommands/-", "value": "echo \"Environment=SYSTEMD_TIMEDATED_NTP_SERVICES=chronyd.service:systemd-timesyncd.service \" >> /lib/systemd/system/systemd-timedated.service"},
{"op": "add", "path": "/spec/template/spec/preKubeadmCommands/-", "value": "systemctl daemon-reload"},
{"op": "add", "path": "/spec/template/spec/preKubeadmCommands/-", "value": "systemctl restart chronyd"}]'

3. Patching the object KubeadmConfigtemplate won't trigger the recreation of the worker nodes and you have to patch the Machine Deployment to trigger the rollout of the worker nodes:
 

kubectl patch machinedeployment MACHINE-DEPLOYMENT --type merge -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"

Validate that the output by running the command "timedatectl status" shows the  NTP service as active from the newly created nodes.

Additional Information

This KB is applicable only for legacy TKG cluster and not classy clusters.