irregular postgres restarts causing issues with workflows in Aria Orchestrator cluster

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Symptoms:

• Orchestrator workflows are failing sporadically
• upon reviewing Aria Automation services with command kubectl get pods -n prelude it is found that a postgres pod shows several restarts
• within /services-logs/prelude/postgres-#/file-logs/liveness.log the following can be observed for the time of the postgres restart:

  01-Jan-2025 01:00:19.934 UTC Replication broken:
  01-Jan-2025 01:00:19.940 UTC Failed to get 'Replication lag' from repmgr node status
  01-Jan-2025 01:00:49.832 UTC Replication lag:  seconds

Environment

Aria Orchestrator 8.18.x cluster

Cause

Due to unconfigured NTP settings the Database replication drifted apart and required a service restart.

This may be a result of a known issue which can occur after installing Aria Automation 8.18.1 Patch 2: Missing NTP settings post upgrade to Aria Automation/Orchestrator 8.18.1 Patch 2

Resolution

Please ensure all Aria Orchestrator appliances are configured with the same NTP server:

Enable Time Synchronization for Automation Orchestrator