How to find which workflows have large tokens associated to them?

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

When browsing large workflow runs in Aria Automation Orchestrator, the UI may return a 502 or 503 response code.

This can cause the Orchestrator service to restart, and a heap dump file may be generated.

Often times, this can be caused by custom workflows, where there may be functions/logic that is storing more information than necessary.

How do we find out which workflows are generating large token sizes so they can be reviewed for correctness or potential errors?

 

Environment

• VMware Aria Automation 8.x
• VMware Aria Automation Orchestrator 8.x

Cause

Large workflow tokens can cause memory allocation issues when the content is being decompressed.

Users may not be able to browse the workflow runs, where the browser returns 502/503 response codes.

Resolution

Solution:

2 of the columns in the vmo_workflowtoken table are id and wfexecutionstack, where:

1. wfexecutionstack is a searchable workflow ID in the UI

2. id corresponds to tokenid in the vmo_workflowtokenstatistics table where where the tokensize information is

 

The steps are to:

1. get all tokenid from vmo_workflowtokenstatistics where the tokensize is greater than 10M

2. use those tokenid as id in the vmo_workflowtoken table to return the wfexecutionstack

3. search in the UI using the wfexecutionstack to find the workflow

 

Please take the usual precautions by taking a non-memory quiesced snapshot of the appliance first.

Then, SSH into the Automation Orchestrator appliance.

Next, run the following commands to connect to the vco-db:

vracli dev psql
\c vco-db

Finally, execute the following command:

select wfexecutionstack from vmo_workflowtoken where id in (select tokenid from vmo_workflowtokenstatistics where tokensize >= 100000000);

Additional Information

Extended explanation with example:

Note: The examples below are from the publicly available VMware Hands On Labs, referencing out-of-the-box workflows.

Using \x to output the data in a readable format, below is an example where there are 4 entries in the vmo_workflowtokenstatistics table, 1 of size 689, and 3 of size 627:

 

To find all the entries of size 628 or greater:

 

We then use that ID to search for the corresponding workflow ID:

 

Finally, that workflow ID can be used to search for the corresponding workflow in the UI:

 

The single-line query to get the list of all workflow IDs using the above criteria would be:

 

Above, we use where id in instead of where id =, as there may be more than one match, and using the latter would return an error (e.g. using a smaller token size of 627 to produce more results):