ESXi host fails to synchronize with NTP server despite successful port connectivity (UDP 123)

Products

• VMware vSphere ESXi

Issue/Introduction

 

• The output of command esxcli system ntp get run on an ESXi host shows  Time Service Enabled: true  but also   Time Synchronized: false.
  

  # esxcli system ntp get
     Enabled: true
     Loglevel: info
     PID: #######
     Runtime Seconds: #######
     Servers: ntp.example.com
     Service Providing Kernel Time: Network Time Protocol
     Time Service Enabled: true
     Time Synchronized: false

  
  
  
• ntpq -p  shows the remote server in .INIT. state with reach 0 and st 16.
  

  remote           refid     st t when poll reach   delay   offset  jitter
  ==============================================================================
  ##.###.#.#      .INIT.     16 u    - 1024    0    0.000   +0.000   0.000

  
  
  
• Netcat commands from the ESXi host to the NTP server indicate the port is open.
  

  # nc -v -u -z -w 3 ntp.example.com 123
  Connection to ntp.example.com 123 port [udp/ntp] succeeded!

   

• Significant time drift may occur over time despite manual clock corrections.
  
  
• The ESXi host's Management Network has a VLAN configured, and testing confirms that NTP packets are received by ESXi but are tagged with a different VLAN. To confirm:
  
  
  1. Run a packet capture on the ESXi host to observe the tagging of incoming NTP traffic by running a packet capture from the ESXi host CLI:  pktcap-uw --ip <NTP_SERVER_IP> --port 123
     
     
  2. Check the output for the VLAN tag value in the captured frames.
     
     
  3. Check the VLAN ID assigned to the Management VMkernel interface via the UI or command line with esxcfg-vswitch -l .
     
     
  4. Verify that the VLAN ID shown for the Management Network is different from the VLAN tag # in the packet capture.

Cause

The issue is caused by a VLAN mismatch between the incoming NTP response packets and the VMkernel interface configuration. While the host can send requests, the returning traffic is tagged with a VLAN ID that does not match the Management VMkernel's VLAN. Consequently, the ESXi TCP/IP stack drops the responses, preventing synchronization.

Resolution

Unless the ESXi host's Management Network has been configured with an incorrect VLAN ID, the fix will need to be done on the customer's physical network.

1. Update the physical switch configuration to ensure that traffic destined for the ESXi Management IP is tagged with the correct Management VLAN ID.
   
   
2. Then, restart the NTP daemon on the host to initiate a fresh handshake:
   
   
   • /etc/init.d/ntpd restart
     
     
   • watch -n 10 "ntpq -p"
     ^ The reach value should begin to increment (1, 3, 7...) and the st (stratum) should change from 16 to a valid lower number.

 

Additional Information

Refer also to:

• (KB 312204) Troubleshooting NTP on ESXi for general NTP analysis and troubleshooting tips
• (KB 311764) VLAN configuration on virtual switches, physical switches, and virtual machines which is helpful in understanding where and how VLANs are set in a vSphere environment.