Workflow Schedules stop with no reason and when Selecting Scheduled followed by selecting any scheduled workflow, the GUI fails to load any content

Products

• VMware Aria Suite

Issue/Introduction

When you create a schedule to run some workflows in VRealize Orchestrator, workflows would stop running and the schedule would have to be recreated again to resume workflows. 

Environment

VMware Aria Automation 8.x

Cause

• VMware Aria Automation Orchestrator uses a token from the authentication provider on behalf of the user whenever a scheduled task is created.
  • Authentication providers will limit the number of tokens a service may receive and when this limit is hit, the tokens are stored but still expire.
• Token renewal periodic task cannot run due to a database deadlock.
• A long running workflow schedule list may overload the web application when a user selects this list of workflow token executions.

Resolution

Prerequisites

• You must back up all VMware Aria Automation or Orchestrator appliances, at the same time - simultaneously for all nodes.
  • If you are taking the snapshots manually, you must start the snapshots of the second and the third node not more than 40 seconds after you start the snapshots for the first node.
  • When you back up the VMware Aria Automation or Orchestrator appliance, disable in-memory snapshots and enable quiescing.

Procedure

Recreate the Schedule:

1. Delete the schedule that fails to load or is no longer triggering at the expected time.
2. Recreate the schedule and associate it with the necessary workflow.

Expired Tokens:

1. Use user credentials rather than tokens (less secure). 
2. Create a day 2 action that would delete and recreate these schedules so that the momentum keeps flowing. 

Database Deadlock:

1. Upload attached .jar file to environment
2. Run command to make it executable chmod a+x "filename.jar"
3. Run the file ./filename.jar