Host commission in SDDC fails with "SDDC Manager is not in sync with NTP server"

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

Host addition and validation fails 

logs on sddc manager located at :/var/log/vmware/vcf/operationsmanager/operationmanager.log shows entries similar to

YYYY-MM-DDTHH:MM:SS DEBUG [vcf_om,xxxxxxxxxxxxxxx,xxxx] [c.v.v.h.c.s.i.CommissionHostsValidator,http-nio-xxx.x.x.x-7300-exec-3] Verifying ntp sync status on SDDC manager using NtpService
YYYY-MM-DDTHH:MM:SS INFO  [vcf_om,xxxxxxxxxxxxxxx,xxxx] [c.v.evo.sddc.common.util.NtpService,http-nio-xxx.x.x.x-7300-exec-3] NTP-UDP-Client timeout 15000
YYYY-MM-DDTHH:MM:SSERROR [vcf_om,xxxxxxxxxxxxxxx,xxxx] [c.v.evo.sddc.common.util.NtpService,http-nio-xxx.x.x.x-7300-exec-3] SDDC Manager is not synced with NTP server
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_om,xxxxxxxxxxxxxxx,xxxx] [c.v.v.h.c.s.i.CommissionHostsValidator,http-nio-xxx.x.x.x-7300-exec-3] SDDC Manager is not in sync with NTP servers
YYYY-MM-DDTHH:MM:SSDEBUG [vcf_om,xxxxxxxxxxxxxxx,xxxx] [c.v.v.h.c.s.i.CommissionHostsValidator,http-nio-xxx.x.x.x-7300-exec-3] Verified ntp sync status of SDDC manager using NtpService
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_om,xxxxxxxxxxxxxxx,xxxx] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-xxx.x.x.x-7300-exec-3] Processing localizable exception SDDC Manager is not in sync with NTP server(s).
YYYY-MM-DDTHH:MM:SS ERROR [vcf_om,xxxxxxxxxxxxxxx,xxxx] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-xxx.x.x.x-7300-exec-3] [984SQ3] SDDC_MANGER_NTP_SERVER_NOT_IN_SYNC SDDC Manager is not in sync with NTP server(s).
com.vmware.evo.sddc.common.core.error.InvalidStateException: SDDC Manager is not in sync with NTP server(s).
        at com.vmware.vcf.hostmanager.comm.controller.v1.HostManagerCommDecommIsController.validatePreConditions(HostManagerCommDecommIsController.java:338)
        at com.vmware.vcf.hostmanager.comm.controller.v1.HostManagerCommDecommIsController.validateHostsOperations(HostManagerCommDecommIsController.java:279)

journal logs in SDDC:

YYYY-MM-DD photon ntpd[1064]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
YYYY-MM-DD photon ntpd[1064]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
YYYY-MM-DD FQDN ntpd[1814]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
YYYY-MM-DD FQDN ntpd[1814]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
YYYY-MM-DD FQDN ntpd[1848]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
YYYY-MM-DD FQDN ntpd[1848]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
YYYY-MM-DD FQDN ntpd[1095]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
YYYY-MM-DD FQDN ntpd[1095]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
YYYY-MM-DD FQDN ntpd[1095]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
YYYY-MM-DDFQDN ntpd[1096]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
YYYY-MM-DD FQDN ntpd[1096]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
YYYY-MM-DD FQDN ntpd[1086]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
YYYY-MM-DD FQDN ntpd[1086]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
YYYY-MM-DD FQDN ntpd[1084]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
YYYY-MM-DD FQDN ntpd[1084]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized

Environment

VCF 5.x

VCF 9.x

Cause

• Wrong NTP server configured
• Time on NTP server is incorrect 
• Any underlying networking issue

Resolution

• Validate the time on NTP server and SDDC manager 
• Restart the NTP server  (Command : systemctl restart ntpd)
• Check the status of the NTP server (Command : systemctl Status ntpd)