VMware Aria Orchestrator is crashing with error "java.lang.OutOfMemoryError: GC overhead limit exceeded"

Products

• VMware Aria Suite

Issue/Introduction

• VMware Aria Orchestrator crashes with error "java.lang.OutOfMemoryError: GC overhead limit exceeded"
• A heap dump (.hprof file) is created in /services-logs/prelude/vco-app/file-logs/
  • There are many causes of the Orchestrator pod crashing and creating a heap dump. This article covers just one possible cause.
• There are millions of records in the vmo_authorizationreference DB table. 

Environment

• VMware Aria Automation Orchestrator 8.x

Cause

• Note there can be many causes of the Orchestrator pod crashing and creating a heap dump
• This article refers to a scenario where we see many stale references returned to this SQL query:
  • SELECT COUNT(1) FROM vmo_authorizationreference
WHERE refobjtype = 'ResourceElement'     
AND refobjid NOT IN (SELECT id FROM vmo_resourceelement)
;

 

Resolution

Before making any database edits, please snapshot all Aria Automation nodes. It is also good practice to take a full database dump

1. Log in to the Orchestrator database:
   • vracli dev psql vco-db
2. Check for stale references:
   • SELECT COUNT(1) FROM vmo_authorizationreference
WHERE refobjtype = 'ResourceElement'     
AND refobjid NOT IN (SELECT id FROM vmo_resourceelement)
;
3. Delete these stale references:
   • DELETE FROM vmo_authorizationreference
WHERE refobjtype = 'ResourceElement'     
AND refobjid NOT IN (SELECT id FROM vmo_resourceelement)
;