Aria Orchestrator UI crashes and vco-app pod restarts when workflows runs exists with large workflow token run sizes

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• When trying to view competed workflow runs in the Orchestrator client its stays spinning and the vco-pod crashes. Running kubectl -n prelude get pods the vco-app pod can be seen restarting with a crashLoopBackOff status.
• The token size for the workflow runs are Megabytes in size. Dashboard -> Usage -> Largest Tokens will show the largest tokens to help identify any offending workflows. Individual token size can also be viewed under Workflow Runs -> Performance.
• Deleting workflow token execution runs for the offending workflows with large execution run token sizes improves stability of GUI
• Warnings regarding content size limits being exceeded are visible in the /var/log/services-logs/prelude/vco-app/file-logs/vco-server-app.log file:

WARN vco [host='vco-app-##########-hbmtf' thread='WorkflowExecutorPool-Thread-13' user='<USERNAME>' org='-' trace='-'] {|__SYSTEM|<USERNAME>:<WORKFLOW_NAME>:########-####-####-####-########5c83:token=########-####-####-####-########6d0e} com.vmware.o11n.service.domain.VmoWorkflowtokencontent - Workflow run with id: ########-####-####-####-########6d0e content size 14254983 exceeds soft limit of 4194304

Environment

VMware Aria Orchestrator 8.x

Cause

The issue is caused by large workflow token run content size. Once the workflow run completes it is stored in the Database, Any client browsing the workflow token will cause this token to be loaded from Database, in vRO Server memory and sent to the client browser.

For example if there are 10 executions of a workflow with a token size of 32MB this will mean transferring 300MB+ of data for the Aria Orchestrator client when browsing through workflow execution history.

Resolution

• To resolve the issue the workflow token run size should be reduced.
  • This can be achieved a number of ways and will be dependent on the design and implementation of the workflows. The size of any inputs, outputs & attributes of the workflow should be optimized. Any non-essential large objects or arrays of objects should be deleted or cleared before the workflow run completes so as not be be contributing to the overall workflow token size
  • Engage your workflow developer for assistance in this regard.
• Another approach is to reduce the number of workflow execution runs stored per workflow, however if the workflow tokens sizes are quite large this may not have much of an impact. By default orchestrator stores 100 workflow execution runs per workflow.
  • To reduce the number of runs stored per workflow follow the steps contained in the Configuring the Workflow Run Properties section of the Orchestrator documentation to reduce the ch.dunes.task.max-workflow-tokens property from the default value of 100.
  • With the Automation Orchestrator 8.18.1 release, the Control Center is removed, hence we will need to do it via CLI
  • To reduce the number of runs stored per workflow run below command
    • vracli vro properties set  --key ch.dunes.task.max-workflow-tokens --value 10

Document for reference : https://techdocs.broadcom.com/us/en/vmware-cis/aria/aria-automation/8-18/vco-installing-and-configuring-8-18/initial-configuration/configuring-automation-orchestrator-with-the-command-line-interface/additional-command-line-interface-configuration-options.html