Provisioning fails with Invalid Refresh Token error in Aria Automation Orchestrator (vRO) REST Workflows

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Provisioning of new virtual machines fails with below error when Aria Orchestrator (vRO) workflows make REST API calls back to the Aria Automation environment
  
  Error fetching access token. Invalid refresh token provided. (Dynamic Script Module name :#461)
  
  
• Restarting the vCO (vRO) pods temporarily resolves the issue, allowing provisioning to succeed for a few days. However, the issue reoccurs after some time. This behavior is typically observed during workflow execution that attempts to reuse an existing authentication token or REST connection.
  
  
• The following error signature is found in /services-logs/prelude/vco-app/file-logs/vco-app.log:
  
  Extensibility triggered task failed. Event ID: ######-####-####-#####. Failure: Extensibility error received for topic compute.allocation.pre, eventId = '#####': [####] SubscriberID: vro-gateway-###, RunnableID: ##### and SubscriptionID: sub_#### failed with the following error: Workflow run [######] completed with error [Error fetching access token. Invalid refresh token provided. (Dynamic Script Module name : ....#script)]

        

Environment

Aria Automation 8.18.1

Cause

The issue is caused by the vCO REST plugin attempting to reuse a closed HTTP stream. When the workflow reuses an existing REST connection, the closed stream results in connection failures and token refresh errors.

Resolution

Broadcom is aware of this issue and plans to address it in an upcoming Aria Automation patch release.

Until the fix is available, apply one of the following workarounds to mitigate the issue:

Workaround 1

set the value of the system property plugin.http_rest.httpclient.connectionpool.tcp_close_after_inactivity_timeout_seconds to 1 using the below command 

vracli vro properties set -k plugin.http_rest.httpclient.connectionpool.tcp_close_after_inactivity_timeout_seconds --value=1

Workaround 2

Modify the REST requests to include an HTTP header that explicitly closes the connection. 

var request = restHost.createRequest(requestType, operationUrl, requestContent, requestContentType);
request.setHeader("Connection", "close");