NTP sync failures when changing existing NTP servers to AWS Time Sync service

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

VMware infrastructure components (vCenter Server, ESXi hosts, and appliances) require accurate time synchronization for cluster stability, authentication, and log correlation.

This article tackles the challenges of maintaining accurate time synchronization during cloud migrations and explains how to resolve NTP connectivity drops when moving to AWS Time Sync service

Recognizing the signs of a failed cloud migration are below:

• ESXi hosts or vCenter Server appliances report "Time Skew" or "Clock Unsynchronized" errors.
• Authentication failures with Active Directory.
• NTP diagnostics (ntpq -p) show a status of INIT or reach value of 0

Environment

• ESXi 8.x
• ESX 9.x
• vCenter 8.x
• vCenter 9.x

Cause

VMware infrastructure components (vCenter Server, ESXi hosts, and appliances) cannot reach the AWS Time Sync Service address (169.254.169.123) because Link-Local addresses are non-routable. Networking protocols prevent these addresses from transiting across routers or tunnels between on-premises sites and an AWS VPC ( Amazon Virtual Private cloud)

Resolution

1. Identify the private IP address of the AWS-hosted Primary Domain Controller (PDC) or a routable NTP server.
2. Configure the on-premises vCenter Server and ESXi hosts to use the routable private IP as the NTP target.
3. Open UDP Port 123 bidirectionally on all firewalls, Security Groups, and Network ACLs between the on-premises site and the AWS VPC(Amazon Virtual Private cloud)
4. For workloads natively running inside the AWS VPC, use the AWS Time Sync Service (169.254.169.123).

Verify synchronization on ESXi by running:

1. Validate Network Connectivity
        a. Execute the vmkping command from the management VMkernel interface.(Typically vmk0)
            
      vmkping -I vmk# <NTP server IP>
    
2. Query NTP Service:
        a. Use the `ntpq` utility to check synchronization status from the ESXi command line to monitor real-time polling:

           watch ntpq -pn

        b. Monitor the output for 30 seconds. A reach value of 377 indicates successful contact over the last eight polls.
        c. If refid column displays INIT, the host has not received a valid response from the NTP server.

If further assistance is required, Contact Broadcom Support.

Additional Information

Using the PDC's private IP allows for standard routing over the existing hybrid connectivity (VPN/Direct Connect). This ensures a consistent and reliable time source while fulfilling the requirement to decommission the Old NTP server without encountering the routing failures associated with Link-Local addresses.

NTP Synchronization Issues on ESXi Hosts

[VMC on AWS] Use Amazon Time Sync Service as an NTP server for compute workloads within VMC SDDC

Troubleshooting NTP on ESX and the ESXi