Workflows show the banner error: "Some data cannot be retrieved. If the problem persists, contact your system administrator. Failed request: <ModuleName>/<ActionName> ... Error getting external data: 502 OK"

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• For example, the F5 plugin workflows "Create Virtual Server" or "Create Pool Member" may show 502 error and not complete.
• This can also affect Service Broker forms, which use vRO to fetch external values.
• The /services-logs/prelude/vco-app/console-logs/vco-server-app.log file contains out of memory errors similar to: 

  java.lang.OutOfMemoryError: Java heap space
  Dumping heap to /var/log/collected-logs/vco_Datestamp_Timestamp_heap_dump.hprof ...
  Heap dump file created [4568732636 bytes in 5.664 secs]
  Terminating due to java.lang.OutOfMemoryError: Java heap space.

• When describing the pods using the kubectl -n prelude get pods command the vco pods show a high number of restarts.
• Checking Memory Consumption on the appliance shows high:

  ps aux --sort=-%mem

• The journal logs /services-logs/journal/systemd.journal may contain an oom-killer message similar to:

  VM Thread invoked oom-killer

• Policy's stopped running without Error-Message

Environment

VMware Aria Automation Orchestrator 8.x

Cause

Where Orchestrator does not have enough heap memory to service requests, the vco-app pod may crash and restart, meaning no response is sent to the form.

Resolution

To avoid the issue either reduce the memory heap usage or increase the Aria Automation Orchestrator heap memory size. If the issue does re-occur after allocating more memory then a heap analysis is recommended.

• To increase the allocated heap memory:
  
  
  • For standalone Aria Automation Orchestrator instances follow the steps contained in How to Scale the Heap Memory Size of the Automation Orchestrator Server.
  • For the Aria Automation Orchestrator instance embedded in the Aria Automation appliance you must increase the VMware Aria Automation profile size through the VMware Aria Suite Lifecycle. For information on supported VMware Aria Automation profiles, see System Requirements.
    * In such cases where the Aria Automation XL deployment profile doesn't satisfy the orchestrator memory requirements, it is recommended to deploy a standalone Orchestrator. Then increase heap memory there as needed, by the above process.
    
    
• To review the the memory usage:
  
  
  • Download the generated heap dump in folder /services-logs/prelude/vco-app/file-logs the filename can be found in /services-logs/prelude/vco-app/console-logs/vco-server-app.log, file ending is .hprof
    (The date/time in the filename is when the service was started before and not to be mislead with the date/time of the heap dump creation, for more details please see here: Mismatch in Aria Orchestrator Heap Dump filename and the date/time of creation.)
  • Follow steps 3 onwards of 4. Analyzing Java Heap Dumps (.hprof files) in Troubleshooting VMware Aria Automation and Aria Orchestrator 8.x application start issues