Aria Orchestrator workflows fail after exactly 1 hour with error "InternalError: Server error : No active session with id [_____]"

Products

• VMware Aria Suite

Issue/Introduction

• Errors seen in Orchestrator main log, /services-logs/prelude/vco-app/file-logs/vco-server-app.log around this workflow failure:
  • InternalError: Server error : No active session with id [<UUID>]
  • Error: Desired state startup failed for: ____ with error: InternalError: Server error : No active session with id [<UUID>]
  • Find in session cache: sessionId: <UUID>; Session not found in cache!
  • Load by session id: sessionId: <UUID>; Session not found!
  • Unable to process pending token operations
• Workflows may hang indefinitely, rather than failing outright
• Some elements in the GUI such as inventory contents may fail to load and keep spinning: in the browser "dev tools" these requests time out without receiving any response.
• From the logs it appears that something is going wrong with delegating the tokens from the UI. Delegation is used in order to persist the access token used in the client in order to be used in the future in the server.

Environment

• VMware Aria Orchestrator 8.x - versions 8.16.2 and later

Cause

• In versions prior to 8.16.2 Orchestrator was using a non-PKCE flow for login. When running a long-running operation the orchestrator server could renew these tokens.
  
• In 8.16.2 the flow was switched to PKCE OAuth2, using a public client id. This in turn caused the server to not have access to the refresh token, since only the browser knows about it.
  
• These later versions use token delegation: when Orchestrator sees a new access token, it tells the identity provider that it wants to use this token for a long running operation and in turn the IDP issues a refresh token.
  
• The issue can occur if this new refresh token was not persisted in the database, only persisted in a local cache. The expiration of this cache is 1h, Orchestrator server tries to find the token in the database, but it fails.

Resolution

This issue will be solved in Aria Automation / Orchestrator 8.18.1 Patch 2 once it is released.

 

Workaround

We can work around the issue by disabling token delegation:

1. Please first take a snapshot on all nodes.
2. Run this command from the SSH session to set the delegation property to false.
   • vracli vro properties set -k "com.vmware.o11n.auth.csp.delegate_tokens" -v "false"

Once Patch 2 is released and installed, you should be able to remove the workaround property so Orchestrator delegates tokens as intended.