systemd-timesyncd is flapping in the Supervisors

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

The systemd-timesyncd process is flapping every 10 seconds on all Supervisor nodes.

# journalctl -u systemd-timesyncd
MM DD hh:mm:33 systemd[1]: Stopping Network Time Synchronization...
MM DD hh:mm:33 systemd[1]: systemd-timesyncd.service: Succeeded.
MM DD hh:mm:33 systemd[1]: Stopped Network Time Synchronization.
MM DD hh:mm:33 systemd[1]: Starting Network Time Synchronization...
MM DD hh:mm:34 systemd[1]: Started Network Time Synchronization.
MM DD hh:mm:34 systemd-timesyncd[3339399]: Initial synchronization to time server <NTP_SERVER>:123 (<NTP_SERVER>).
MM DD hh:mm:43 systemd[1]: Stopping Network Time Synchronization...
...(looping every 10 seconds)

 

Too many debug messages are logged in sync.log on all Supervisor nodes.

# /var/log/update-controller/sync.log
YYYY-MM-DDTHH:mm:ss.sssZ DEBUG authentication_setting: Synced root password to the latest configuration.
YYYY-MM-DDTHH:mm:ss.sssZ DEBUG authentication_setting: Synced root password to the latest configuration.
YYYY-MM-DDTHH:mm:ss.sssZ DEBUG authentication_setting: Synced root password to the latest configuration.

 

chpasswd command is kicked every 10 seconds on all Supervisor nodes.

# journalctl --since today
MM DD hh:mm:ss ####### audit[2960304]: SYSCALL arch=c000003e syscall=82 success=yes exit=0 a0=7fc8414abea7 a1=7fc8414abe31 a2=7fc841a4ea40 a3=4df2 items=5 ppid=636 pid=2960304 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="chpasswd" exe="/usr/sbin/chpasswd" subj=unconfined key="shadow"

Environment

VMware vSphere Kubernetes Service - Earlier than vCenter 8u3e (8.0.3.00500)

Cause

The update-controller in the Supervisor sync loop is stuck in an infinite loop.

Resolution

Temporary Workaround

1. SSH to the Supervisor

2. Backup the target file

cp /usr/lib/vmware-wcp/update-controller/sync.py ~

3. Comment out the line number near 516-521, depending on the Supervisor version

chmod 600 /usr/lib/vmware-wcp/update-controller/sync.py

vi /usr/lib/vmware-wcp/update-controller/sync.py
    520         # Sync if domain changes
    521         #retry = self.sync_sso_domain_change(messages) or retry # <-----  comment out this code

4. Restart wcp-sync process

systemctl restart wcp-sync
systemctl status wcp-sync

5. Repeat this operation for the remaining 2 Supervisors.

 

Permanent Solution

Upgrade to vCenter 8U3e.