Task Execution Delay in VMware Cloud Director Due to Time Synchronisation Issues

Products

• VMware Cloud Director

Issue/Introduction

Cloud Director task execution experiences multiple delays of approximately 9-10 minutes.

During these delays, Cloud Director is logging "Retrieving cached security context for user" during task polling in the vcloud-container.debug.log

Environment

VMware Cloud Director

Cause

• VCD's Activity Framework schedules tasks based on the last timestamp recorded in the database.
• A time synchronisation issue between the vCD cell and the database caused incorrect polling logic.
• The vCD cell interpreted the future timestamps incorrectly, assuming tasks were not ready for execution.
• The default polling interval of 4 minutes further contributed to delays by waiting until the future timestamp was reached.

Resolution

• Ensure all VCD cells and database servers are synchronised with a reliable NTP source.
• Verify and correct time synchronisation discrepancies across vCD, DB, and NSX components.
• Monitor VCD's activity framework behavior to confirm polling and execution times are within expected thresholds.
• Review logs across all affected components to ensure consistent timestamp reporting.

Additional Information

How NTP Works in VMware Cloud Director.

VMware Cloud Director appliances rely on NTP to maintain accurate system time across all nodes in a multi-appliance setup.

NTP synchronises time with an external time source, preventing drift - gradual time deviation due to system clock inaccuracy.

If an Appliance's time is out of sync, it can cause authentication failures, inconsistent logging, and delays in distributed system coordination.
    

Impact of One Appliance Being One Hour Behind

If one appliance drifts significantly, its time-sensitive operations such as token expiry, message processing timestamps, or database transactions become inconsistent.

VMware Cloud Director uses a clustered environment where each node must have synchronised time. A one-hour drift means that:

• Logs and event timestamps become inconsistent.
• Scheduled jobs might fail or execute at incorrect times.
• SSL/TLS certificates, session tokens, and API requests may be rejected due to mismatched timestamps.
• Services such as PostgreSQL and Artems, which rely on consistent timestamps, start experiencing delays.