Aria Automation Orchestrator service regular restarts after installing Patch 5 (Aria Automation Patch 4)

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• After installing Patch 5 for external Orchestrator (Aria Automation Patch 4 for embedded Orchestrator) the service keeps restarting and creating heap dump files.
• This can be observed by:
  • Monitoring service restarts from the appliance shell:

    kubectl -n prelude get pods -l app=vco-app -w

  • Reviewing logs:

    grep "Dumping heap" services-logs/prelude/vco-app/file-logs/vco-server-app.log -C 2

    • In /services-logs/prelude/vco-app/console-logs/vco-server-app.log, you will see OOM error:
      
  • After enabling DEBUG logging, you will see that the CacheUpdate version is not advancing:
    
  • An advancing CacheUpdate session would look like this:
    
  • The vco pods may face Startup/Liveness/Readiness probe timeouts, with messages such as these seen in kubernetes events:
    • Startup probe failed: Get "http://10.##.##.##:8280/vco/api/healthstatus?startupProbe=true": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    • Liveness probe failed: Get "http://10.##.##.##:8280/vco/api/health/liveness": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    • Readiness probe failed: Get "http://10.##.##.##:8280/vco/api/health/readiness": dial tcp 10.##.##.##:8280: connect: connection refused

Environment

• Aria Automation Orchestrator 8.18.1 Patch 5
• Aria Automation 8.18.1 Patch 4

Cause

An issue was identified with the Orchestrator vSphere plugin handling inventory updates in larger environments. When vCenter attempts to retrieve a large list of objects, it uses a "paging" process to handle the data in chunks. However, a logic error prevents the system from advancing to the next chunk of data. Instead, it repeatedly requests the same information, causing the data response to grow uncontrollably (up to 1GB per session) until the server runs out of memory.

Resolution

Download the attached updated vSphere plugin for Aria Automation Orchestrator version 8.2.2 (Build 25275217).

Installation:

1. If the Orchestrator web client does not open due to constant restarts then set the following system property via command line: 

   vracli vro properties set -k "com.vmware.o11n.plugin.vsphere.pc.wait_options.max_object_updates" -v "50"

   * This should remediate the restarts but if necessary the value can be reduced to 20.
   
   
2. Validate that the property has been applied by running:

   vracli vro properties

3. Install the plugin following document - Install an Automation Orchestrator plug-in
   
   
4. Once the plugin has been upgraded remove the system property: 

   vracli vro properties remove -k "com.vmware.o11n.plugin.vsphere.pc.wait_options.max_object_updates"

•  

If this does not resolve the issue then a heap dump analysis is required.

• In situations where a lot of heap dumps are generated due to frequent service crashes it may happen that the created heap dump files are incomplete. Disk space is required for the service to create a full heap dump:

1. 1. Clean up all heap dump's *.hprof from folder /services-logs/prelude/vco-app/file-logs/, by either downloading or deleting them.
   2. Monitor the service and wait for a new crash, you can use this command: kubectl -n prelude get pods -l app=vco-app -w
   3. Once the service crashed, please collect a log bundle and the heap dump file *.hprof from folder /services-logs/prelude/vco-app/file-logs/
      * to create a log bundle from the appliance navigate to the home folder (cd) and run vracli log-bundle

Additional Information

Troubleshooting VMware Aria Automation and Aria Orchestrator 8.x application start issues