Intermittent error "Some form data cannot be retrieved" when loading forms from Aria Automation backed by Orchestrator actions or workflows

Products

• VMware Aria Suite

Issue/Introduction

Users encounter intermittent errors when loading Orchestrator forms through VMware Aria Automation blueprints. The error message indicates that some form data cannot be retrieved due to failed requests. This problem occurs inconsistently, with some users able to load forms on certain days and others unable to load them.

 The request did not return a response.
There was a problem retrieving form evaluation context: Http failure response for https://EXAMPLE.FQDN.COM/vro/forms/UUID############/context?projectId=: 403 OK

Environment

VMware Aria Automation 8.x

VMware Aria Automation Orchestrator 8.x

Cause

The issue arises from frequent pod restarts, which lead to inconsistent service behavior. The services fail to stabilize fully after each restart, leading to intermittent failures when retrieving form data. The lack of full recovery after restarts results in failed API calls for form data retrieval. This is common when Aria Automation is rebooted improperly.

This issue can also be caused by workloads with a high amount of memory usage that leads to the vco-app to crash with out of memory errors: Workflows show the banner error: "Some data cannot be retrieved. If the problem persists, contact your system administrator. Failed request: <ModuleName>/<ActionName> ... Error getting external data: 502 OK"

Resolution

Prerequisites

• You must back up all VMware Aria Automation or Orchestrator appliances, at the same time - simultaneously for all nodes. 
  • If you are making the snapshots manually, you must start the snapshots of the second and the third node not more than 40 seconds after you start the snapshots for the first node.
  • If the quiesced state was not achieved for all 3 nodes within the ~40 seconds time frame, the following strings will be found in the logs: "Freeze synchronization failed" and "Sync failed, making inconsistent snapshot".
    Run the following command from one of the nodes to filter for all vmtoolsd messages: journal ctl --identifier=vmtoolsd
  • When you back up the VMware Aria Automation or Orchestrator appliance, disable in-memory snapshots and enable quiescing.

Procedure

 

1. Delete kube-system namespace to clear IP caching and reset the Age. Follow the instructions in the provided Knowledge Base article:
   Aria Automations Requests stuck on Initialization

2. Restart services on all nodes:

   • SSH into ApplianceFQDN1 and run:

      

     /opt/scripts/deploy.sh --shutdown

   • Wait for it to complete and then SSH into ApplianceFQDN2 and ApplianceFQDN3.

   • Run the reboot command on each appliance.

     • Monitor the reboot process via the vSphere console and wait for the PhotonOS splash screen before starting services.

   • Once the nodes reboot, run the following command on each node to restart services:

      

     /opt/scripts/deploy.sh

3. Verify service stability:
   Once the Prelude deployment is successful and services are restored, attempt to load forms and verify if the issue is resolved.