vco-pods crash with out of memory errors when workflows generate a high number of audit log events in VMware Aria Automation Orchestrator

Products

• VMware Aria Suite

Issue/Introduction

 

• The /services-logs/prelude/vco-app/console-logs/vco-server-app.log file contains out of memory errors similar to:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to /usr/lib/vco/app-server/../app-server/logs/vco_Datestamp_Timestamp_heap_dump.hprof ...
Heap dump file created [4568732636 bytes in 5.664 secs]
Terminating due to java.lang.OutOfMemoryError: Java heap space.

• When describing the pods using the kubectl -n prelude get pods command the vco pods show a high number of restarts:

• When querying the vco database, the vmo_clusterauditlog table contains a large count of audit logs for individual workflow execution IDs.

Environment

VMware Aria Orchestrator 8.x

Cause

The issue can occur when workflows generate an abnormally large amount of audit logs.

Resolution

To determine if you are hitting this particular issue you will need to query the vCO database vmo_clusterauditlog table to see if there are any workflow execution runs that have generated a large amount of audit logs.

Note: before proceeding take a snapshot of the Aria Orchestrator appliance(s).
 

1.  SSH to the Aria Orchestrator  appliance and login as root user.
2. To connect to the vCO database:
   vracli dev psql vco-db
   Type yes when prompted.
3. To count audit logs per workflow run execute the select query:
   SELECT COUNT(eventdata) AS occurrences, eventdata
   FROM vmo_clusterauditlog
   GROUP BY eventdata
   ORDER BY COUNT(eventdata) DESC;
4. If the count returned for any workflow execution id is higher than 10,000 consider removing them:
   DELETE FROM vmo_clusterauditlog
   WHERE eventdata
   IN ('execution-id-1', 'execution-id-2', ...);
   
   Replace  execution-id-1 & execution-id-2 with the IDs identified in step 3.

If you see out of memory errors but no high count returned from query 3 above then consider increasing the default Aria Orchestrator Java heap memory