Aria Automation Orchestrator "vco-app" Pod crashes due to Out of Memory Error

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

You will notice that vco-app pod will crash abruptly due to Out of Memory errors.

You may notice similar error trace in the logs.

DD-MM-YYYY TestvRA.example.com kubelet[#####]: ##### ##:##:##.##### ##### scope.go:###] "RemoveContainer" containerID="################################################################"
DD-MM-YYYY TestvRA.example.com kernel: WorkflowExecuto invoked oom-killer: gfp_mask=0x###(GFP_KERNEL), order=0, oom_score_adj=###
DD-MM-YYYY TestvRA.example.com kernel: CPU: 0 PID: ###### Comm: WorkflowExecuto Not tainted #.10.###-2.ph4 #1-photon
DD-MM-YYYY TestvRA.example.com kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS X.00 ##/##/####
DD-MM-YYYY TestvRA.example.com kernel: Call Trace:
DD-MM-YYYY TestvRA.example.com kernel: dump_stack########

Environment

Aria Automation 8.18.x

Cause

The issue is caused by concurrency issues within the Active Directory (AD) plugin and the handling of connection pools in custom workflows.

Workflows will invoke AdHost.close(), which internally closes the shared connection pool while it's still being used by other objects or concurrent sessions. This will result in "connection pool has been closed" failures.

Resolution

This is one of the corner case issue and needs a custom build of AD Plugin.

Apart from the custom build plugin, couple of parameters needs to be tweaked.

Please contact Broadcom Support in case you encounter the same issue.