Commission Hosts task on SDDC Manager fails with error "SDDC Manager is not in sync with NTP server"

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• On the SDDC Manager, commissioning hosts fails at Host Addition and Validation task with error "SDDC Manager is not in sync with NTP server".

  
  
  
• On the SDDC Manager, in /var/log/vmware/vcf/operationsmanager/operationsmanager.log:

YYYY-MM-DDTHH:MM:SS DEBUG [vcf_om,############,f###] [c.v.v.h.c.s.i.CommissionHostsValidator,http-nio-127.0.0.1-7300-exec-8] Verified ntp sync status of SDDC manager using NtpService
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_om,############,f###] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7300-exec-8] Processing localizable exception SDDC Manager is not in sync with NTP server(s).
YYYY-MM-DDTHH:MM:SS ERROR [vcf_om,############,f###] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7300-exec-8] [A7LIDT] SDDC_MANGER_NTP_SERVER_NOT_IN_SYNC SDDC Manager is not in sync with NTP server(s).
com.vmware.evo.sddc.common.core.error.InvalidStateException: SDDC Manager is not in sync with NTP server(s).


• Checking the ntpd (Network Time Protocol Daemon) displays the below error messages:
  
  systemctl status ntpd

ntpd.service - Network Time Service
     Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; preset: disabled)
     Active: inactive (dead) since <Day> YYYY-MM-DD HH:MM:SS UTC
   Duration: 17h 59min 18.291s
       Docs: man:ntpd
   Main PID: #### (code=exited, status=0/SUCCESS)
        CPU: 3.549s

<Month> <date> HH:MM:SS <SDDC_MANAGER_FQDN> ntpd[####]: duplicate or replay: org 0x########.######## does not match 0x0.00000000 from server@<IP_Address>
<Month> <date> HH:MM:SS <SDDC_MANAGER_FQDN> ntpd[####]: duplicate or replay: org 0x########.######## does not match 0x0.00000000 from server@<IP_Address>
<Month> <date> HH:MM:SS <SDDC_MANAGER_FQDN> ntpd[####]: duplicate or replay: org 0x########.######## does not match 0x0.00000000 from server@<IP_Address>
<Month> <date> HH:MM:SS <SDDC_MANAGER_FQDN> ntpd[####]: duplicate or replay: org 0x########.######## does not match 0x0.00000000 from server@<IP_Address>
<Month> <date> HH:MM:SS <SDDC_MANAGER_FQDN> ntpd[####]: duplicate or replay: org 0x########.######## does not match 0x0.00000000 from server@<IP_Address>


• The ICMP ping from the SDDC Manager to the NTP server has duplicate responses.
  
  Example:
  
  [ ~ ]# ping <NTP-server-IP>
PING #.#.#.# (#.#.#.#) 56(84) bytes of data.
64 bytes from #.#.#.#: icmp_seq=1 ttl=60 time=58.1 ms
64 bytes from #.#.#.#: icmp_seq=1 ttl=123 time=75.3 ms (DUP!)
64 bytes from #.#.#.#: icmp_seq=2 ttl=123 time=40.6 ms
64 bytes from #.#.#.#: icmp_seq=2 ttl=60 time=57.1 ms (DUP!)
64 bytes from #.#.#.#: icmp_seq=3 ttl=123 time=40.0 ms
  
  

Environment

VMware Cloud Foundation 9.x

Cause

• An internal network routing or switching anomaly causes duplicate packets to be delivered to the SDDC Manager.
• The ntpd service detects these duplicate/replay packets, which destabilizes the time synchronization process and ultimately causes the service to exit, failing the host commission prerequisite checks.

Resolution

• Identify and remediate the underlying network packet duplication within the environment's infrastructure with assistance from the internal networking team.
• Resolving the underlying network packet duplication restores SDDC Manager NTP synchronization, fulfilling the prerequisite for the Operations Manager to successfully complete the host commissioning workflow.