Git Pull or Push operation fails with a Transaction timeout expired error in VMware Aria Automation Orchestrator

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• You are attempting Git Push or Pull operations from within VMware Aria Automation Orchestrator.
• You receive an error similar to the below:

  Transaction timeout expired.

Environment

VMware Aria Automation Orchestrator 8.x
VMware Aria Automation 8.x

Cause

• The amount of time a Pull or Push operation may take is related to the number of local changes that are currently available or have been pushed.
• The larger the number of changes, the more time it may take to complete.
  • When there is a local change, or a remote change (meaning it is incoming from the integrated remote repository) that is a deletion of an Orchestrator object, all of the local changes are reapplied (the Orchestrator object gets updated in the database with the corresponding local change from the repository) to make sure that all of the objects' contents are up to date.
• The default database transaction timeout set is 600 seconds.
  • For a Git Pull or Push operation to fail with a Transaction timeout expired error it means that 600 seconds have passed since the start of the operation, reaching the maximum configured timeout.

Resolution

VMware is aware of this issue. See the Workaround section below for additional information.

Workaround:

Prerequisites

• You have root username and password.
• You have SSH access to the appliances.
• You have backups of the VMware Aria Automation Orchestrator 8.x appliance(s)
  • You must back up all VMware Aria Automation Orchestrator appliances, at the same time - simultaneously for all nodes.
  • If you are taking the snapshots manually, you must start the snapshots of the second and the third node no more than 40 seconds after you start the snapshots for the first node.
  • If VMware Automation Orchestrator is embedded within the VMware Aria Automation appliance, the backup is performed as part of the VMware Aria Automation backup.

Procedure

The database transaction timeout is configurable and may be adjusted using the below steps:

1. SSH into the VMware Aria Automation Orchestrator appliance.
2. In the Bash terminal, run the following command to change the database transaction timeout value:

   vracli vro properties set -k com.vmware.o11n.db.transaction.timeout -v 1200

Note: The above command sets the database transaction timeout to a value of 1200 seconds but it can be any value - set it higher, if needed.

4. Wait a few minutes for the Orchestrator pod to start and enter a Running state. You may monitor the services starting using the following command:

   kubectl -n prelude get pods -w