Clock time drift observed in cloud foundry.

Products

• VMware Tanzu Application Service

Issue/Introduction

Application team have noticed clock time drift issue for their services response time. 

During troubleshooting it was discovered that the diego cells were out of time sync 

 

Environment

TAS X.X

Cause

NTP specified servers lost access/connectivity to the their higher level NTP server (Lost Strata)

If an NTP server loses its stratum, it typically signifies that the server has lost synchronisation with its time source or is unreachable. 

This can lead to clients relying on that server for time synchronisation experiencing inaccurate or inconsistent time values. In such cases, the server's stratum will likely change to 16, indicating it's not a viable time source.  

Resolution

Restoring access/connectivity to the top level NTP servers restores the time synchronisation.

To determine the root cause  follow https://broadcomcms-software.wolkenservicedesk.com/external/article?articleNumber=298645   to determine if other factors can play a role 

Additional Information

Following above KB taking packet capture the issue could be spotted from the NTP Server response:

Frame 28: 90 bytes on wire (720 bits), 90 bytes captured (720 bits)

User Datagram Protocol, Src Port: 123, Dst Port: 35529
Network Time Protocol (NTP Version 4, server)
    Flags: 0xe4, Leap Indicator: unknown (clock unsynchronized), Version number: NTP Version 4, Mode: server
        11.. .... = Leap Indicator: unknown (clock unsynchronized) (3)
        ..10 0... = Version number: NTP Version 4 (4)
        .... .100 = Mode: server (4)
    [Request In: 27]
    [Delta Time: 0.001392000 seconds]
    Peer Clock Stratum: unspecified or invalid (0)
    Peer Polling Interval: 6 (64 seconds)
    Peer Clock Precision: 0.000002 seconds
    Root Delay: 0.000000 seconds
    Root Dispersion: 0.000000 seconds
    Reference ID: Unidentified reference source ''
    Reference Timestamp: Apr 26, 2025 10:52:17.206876124 UTC
    Origin Timestamp: Jan 13, 2089 00:20:08.625100633 UTC
    Receive Timestamp: May  7, 2025 08:12:08.619482444 UTC
    Transmit Timestamp: May  7, 2025 08:12:08.620206839 UTC

Compared with working data:

    Leap indicator:  (0), Stratum 2 (secondary reference), poll 10 (1024s), precision -23
    Root Delay: 0.040924, Root dispersion: 0.005950, Reference-ID: 0xXXXXX304
      Reference Timestamp:  3955621211.107362440 (2025-05-07T15:40:11Z)
      Originator Timestamp: 2863520540.845207968 (1990-09-28T14:22:20Z)
      Receive Timestamp:    3955621334.094524586 (2025-05-07T15:42:14Z)
      Transmit Timestamp:   3955621334.094722443 (2025-05-07T15:42:14Z)
        Originator - Receive Timestamp:  +1092100793.249316618
        Originator - Transmit Timestamp: +1092100793.249514474