Automation Orchestrator pods continuously restart with no obvious heap dumps or errors

Products

• VMware Aria Suite

Issue/Introduction

• Orchestrator pods are restarting frequently
• There are no heap dumps
• There are no OutOfMemory Exceptions in the server logs
• Journal logs show oom-killer records
• Garbage collector logs show allocation failures

Environment

VMware Aria Automation Orchestrator 8.13 and later

Cause

 There isn't enough non-heap memory for the garbage collector to work properly.

Resolution

Prerequisites

Take a snapshot of your environment.

Procedure

1. Edit the resource metrics file in your custom profile with the desired memory values.
   vi /etc/vmware-prelude/profiles/custom-profile/helm/prelude_vco/90-resources.yaml
2. Ensure that the serverMemoryRequest is at least 50% bigger than serverJvmHeapMax and that serverMemoryLimit is at least 2G bigger than serverMemoryRequest.
   1. In case serverMemoryRequest cannot be enlarged, decrease the serverJvmHeapMax to about 60% of it.
3. Run /opt/scripts/deploy.sh to restart the system.