NotAuthorizedException in Aria Orchestrator causing the vco-app pods to crash and restart, creating heap dumps

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Users encounter 503 Service Unavailable errors when accessing Aria Automation Orchestrator.
• On the appliance, vco-app pods crash, potentially filling the logs disk space with *.hprof files in /services-logs/prelude/vco-app/file-logs/.
• Reviewing the /services-logs/prelude/vco-app/file-logs/vco-server-app.log reveals the following error:
• ERROR vco [host='vco-app-xxxx' thread='WorkflowExecutorPool-Thread-x' user='<USER_ID>'] {|__SYSTEM|<USER_ID>:<WORKFLOW_NAME>:<UUID1>:token=<UUID2>} ch.dunes.scripting.server.script.DynamicFunction - User LDAP-USER-['<USER_ID>'] - <DOMAIN>\<USER_ID> doesnt have required rights for calling {1} method
ch.dunes.util.NotAuthorizedException: User LDAP-USER-['LECLERMI2'] - <DOMAIN>\<USER_ID> doesnt have required rights for calling {1} method
        at com.vmware.o11n.service.security.AccessRightsInterceptor.checkRoleRestrictions(AccessRightsInterceptor.java:199) ~[o11n-services-server-8.18.1.jar:?]
        at [...]
• This error typically points to a specific workflow, <WORKFLOW_NAME>, failing during execution due to an apparent permissions issue for the executing user <USER_ID>

Environment

VMware Aria Orchestrator 8.x

Cause

The vco-app service attempts to resume or execute a multiple workflows in parallel which causes them to fail. The repeated failure during the Orchestrator checkpointing process triggers a pod crash loop.

 

Resolution

To resolve this issue, identify and address the authorization failure within the specific workflow.

1. Identify the Workflow: Locate the workflow execution ID and the user mentioned in the vco-server-app.log error trace.
2. Validate Permissions: Ensure the user has the required roles within Aria Automation Identity and Access Management and Orchestrator Groups/Roles.
3. Clear Stale Executions: If the pod is stuck in a crash loop due to a specific execution:
   
   1. Access the Orchestrator UI (if possible) and cancel the hung workflow.
   2. If the UI is inaccessible, it can be cancelled from the CLI: see vracli vro cancel -h for details
   3. If this still fails, consider scaling the vco-app deployment to 0 and back to 1 to break the immediate loop, though the workflow may attempt to resume.
4. If the crash loop persists, contact Broadcom Support referencing this article (#445823)
   
   • Creating and managing Broadcom support cases