Aria Automation can't communicate with Aria Automation Orchestrator from time to time

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• sporadic deployment failure
• example error: 

  Failed to process ebs event: SubscriberID: vro-gateway-123456, RunnableID: EXECUTION-RUN-GUID and SubscriptionID: sub_12345 failed with the following error: com.vmware.automation.vro.gateway.common.RemoteVroGatewayException: Automation Orchestrator Gateway received error 404 NOT_FOUND for request: /sessions/exchange error: {"timestamp":"TIMESTAMP","path":"/session-service/api/sessions/exchange","status":404,"error":"Not Found","message":"404 NOT_FOUND \"Session token not found\"","requestId":"12345-123","@type":"org.springframework.web.server.ResponseStatusException"}

• within /services-logs/prelude/tango-vro-gateway-app/file-logs/tango-vro-gateway-app.log the following error can be observed:

  <TIMESTAMP> ERROR tango-vro-gateway [host='tango-vro-gateway-app-123456-12345' thread='reactor-http-epoll-3' user='provisioning-123456789' org='<ORG-GUID>' trace='<TRACE-GUID>' parent='<GUID>' span='<GUID>'] c.v.a.v.g.i.s.c.SessionServiceClient.lambda$exchange$3:97 - Cannot fetch access token for userId: USERACCOUNT: tokenId: <TOKEN> orgId: <ORG-GUID>. Error:
          com.vmware.automation.vro.gateway.common.RemoteVroGatewayException: Automation Orchestrator Gateway received error 404 NOT_FOUND for request: /sessions/exchange error: {"timestamp":"<TIMESTAMP>","path":"/session-service/api/sessions/exchange","status":404,"error":"Not Found","message":"404 NOT_FOUND \"Session token not found\"","requestId":"123456-123","@type":"org.springframework.web.server.ResponseStatusException"}
                  at com.vmware.automation.vro.gateway.common.RemoteVroGatewayException.lambda$handleErrorEx$0(RemoteVroGatewayException.java:39) ~[tango-vro-gateway.jar:na]
                  Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
          Error has been observed at the following site(s):
                  *__checkpoint ⇢ 404 from POST http://project-service.prelude.svc.cluster.local:8080/session-service/api/sessions/exchange [DefaultWebClient]

Environment

clustered instance of Aria Automation 8.x

Resolution

The issue can occur if there is a time drift between configured Aria Automation appliances.

To review the configured time the following command can be used: vracli cluster exec -- bash -c 'hostname; date'

It is recommended to configure the same NTP server for each Aria Automation appliance, to do so please refer to the documentation:

How do I enable time synchronization of VMware Aria Automation`