Approval-service on vRA 8.x fails with status of CrashLoopBackoff and will not go to Running state or ABX endpoint registration failed for non default tenants

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:
1. Approval-service pod on vRA 8.x is stuck at CrashLoopBackoff state and will not go to Running state

In the approval service logs, you see error similar to:

 org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request: [{"timestamp":"<Date>T16:51:28.167+0000","path":"/event-broker/api/subscriptions","status":400,"error":"Bad Request","message":"22086-Not allowed to change current org id '<Org_Id> (490 bytes)]

Command to get see approval service log:
    i) Find out approval service pods - kubectl get pods -l app=approval-service-app -n prelude
    ii) kubectl logs <any of the approval service pod name - outcome of previous command)> -n prelude


2. vRealize Automation deployment fails on ABX endpoint registration in non default tenants with:
* In the deployment log, you see error similar to:

 curl: (22) The requested URL returned error: 500 Internal Server Error
 /opt/scripts/register_abx_endpoint.sh: line 40: [: : integer expression expected
 Register ABX endpoint in org with ID: <Org_id>
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
 100 264 0 0 100 264 0 392 -::- -::- -::- 392
 curl: (22) The requested URL returned error: 500 Internal Server Error
 Deployment failed. Collecting log bundle
 
* In the provisioning service logs, you see error similar to:

 <Date>T17:53:07.399Z [priority='ERROR' thread='reactor-http-epoll-7' user='provisioning-gXrQqJpWzxR2zpBz' org='<Org_Id>' context='<Id>' parent='' token='>Token_id>'] o.s.b.a.w.r.e.AbstractErrorWebExceptionHandler.error:122 - [xxxxxxxx-xxxxx]  500 Server Error for HTTP GET "/provisioning/mgmt/endpoints?enumerate&external&$filter=(endpointType%20eq%20%27abx.endpoint%27)"

com.vmware.automation.spring.webflux.platform.client.service.exception.WebClientServiceResponseException: ClientResponse has erroneous status code: 400 Bad Request. WebClientServiceResponseException.ErrorDetails(timestamp=null, path=null, type=null, errorCode=0, messageKey=null, messageArguments=null, causeMessage=null)

 Command to get see provisioning service log:
    i) Find out provisioning service pods - kubectl get pods -l app=provisioning-service-app -n prelude
    ii) kubectl logs <any of the provisioning service pod name - outcome of previous command)> -n prelude

* In the identity service logs, you see error similar to:

 <Date> 09:10:09.691+0000 ERROR 14 --- [or-http-epoll-1] v.i.c.RestResponseEntityExceptionHandler : Handling bad request exception: Org IDs from token, host or/and request do not match.

java.lang.IllegalArgumentException: Org IDs from token, host or/and request do not match.
<Date> 09:10:09.693+0000 INFO 14 --- [or-http-epoll-1] reactor.netty.http.server.AccessLog : <IP> - - [<Date>:09:10:09 +0000] "GET /csp/gateway/am/api/orgs/<Org_Id> HTTP/1.1" 400 241 8080 285 ms

 Command to get see identity service log:
    i) Find out identity service pods - kubectl get pods -l app=identity-service-app -n prelude
    ii) kubectl logs <any of the identity service pod name - outcome of previous command)> -n prelude

Environment

VMware vRealize Automation 8.x
VMware vRealize Automation 8.1.x

Cause

Clients are treated as users in Enterprise Group Role Assignment thus when assigning a role to ALL USERS group. This changes the roles of the clients and context_name property in the clients token.

Resolution

This issue is resolved in vRealize Automation 8.3 and later.

Workaround:
Go to Identity & Access Management -> Enterprise Groups tab in vRealize Automation - Cloud Services Console .

 {vRA FQDN}/csp/gateway/portal/#/consumer/usermgmt/ad-groups

Remove all organisation and services roles added to ALL USERS enterprise group.

Note: VMware recommends not to use ALL USERS enterprise group to assign roles to users. Use either other local groups or groups from the identity provider (AD, OpenLDAP, etc.) synced in vIDM.