Workflows show the banner error: "Some data cannot be retrieved. If the problem persists, contact your system administrator. Failed request: <ModuleName>/<ActionName> ... Error getting external data: 502 OK"

Products

• VMware Aria Suite

Issue/Introduction

• For example, the F5 plugin workflows "Create Virtual Server" or "Create Pool Member" may show 502 error and not complete.
• This can also affect Service Broker forms, which use vRO to fetch external values.

• The /services-logs/prelude/vco-app/console-logs/vco-server-app.log file contains out of memory errors similar to:

  java.lang.OutOfMemoryError: Java heap space
Dumping heap to /usr/lib/vco/app-server/../app-server/logs/vco_Datestamp_Timestamp_heap_dump.hprof ...
Heap dump file created [4568732636 bytes in 5.664 secs]
Terminating due to java.lang.OutOfMemoryError: Java heap space.

• When describing the pods using the kubectl -n prelude get pods command the vco pods show a high number of restarts.

   

• The journal logs /services-logs/journal/systemd.journal  may contain an oom-killer message similar to:

  VM Thread invoked oom-killer

Environment

VMware Aria Automation Orchestrator 8.x

Cause

Where Orchestrator does not have enough heap memory to service requests, the vco-app pod may crash and restart, meaning no response is sent to the form.

Resolution

Increase the Aria Automation Orchestrator heap memory size:

- For standalone Aria Automation Orchestrator instances follow the steps contained in How to Scale the Heap Memory Size of the Automation Orchestrator Server.

- For the Aria Automation Orchestrator instance embedded in the Aria Automation appliance you must increase the VMware Aria Automation profile size through the VMware Aria Suite Lifecycle. For information on supported VMware Aria Automation profiles, see System Requirements.


In such cases where the Aria Automation XL deployment profile doesn't satisfy the orchestrator memory requirements, it is recommended to deploy a standalone Orchestrator. Then increase heap memory there as needed, by the above process.