vRA 8.1 multi tenancy environment deployment of identity-service data fails with 403

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:
The script responsible for initial vRA deployment /opt/scripts/deploy.sh fails with a 403 error

=========================
[2020-07-18 09:48:46.575+0000] Populating initial identity-service data
=========================

vaconfig.prelude.vmware.com/prelude-vaconfig patched (no change
 % Total % Received % Xferd Average Speed Time Time Time Current
                 Dload Upload Total Spent Left Speed
^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0^M100 29 0 0 100 29 0 24 0:00:01 0:00:01 --:--:-- 24^M100 29 0 0 100 29 0 15 0:00:01 0:00:01 --:--:-- 15
curl: (22) The requested URL returned error: 403 Forbidden
Deployment failed. Collecting log bundle ...


vIDM accesscontrol-service.log displays:

2020-07-18 09:48:46,474 GMT WARN  vidm.lab:accesscontrol (ForkJoinPool-4-worker-0) [TENANT1;<UUID>;<IP>;<UUID>] com.vmware.vidm.common.interservice.ServiceTokenProviderGateway - Invalid credentials for service token. Http Response code: 403
2020-07-18 09:48:46,474 GMT WARN  vidm.lab:accesscontrol (ForkJoinPool-4-worker-0) [TENANT1;<UUID>;<IP>;<UUID>] com.vmware.vidm.accesscontrol.gateway.
UserGroupServiceGateway - UserGroup service call failed com.vmware.vidm.common.interservice.exception.InvalidClientCredentialsException: Invalid credentials used to get service token
        ...
Caused by: ScForbiddenException[Operation: GET -> https://master.vidm.lab:443/SAAS/jersey/manager/api/scim/Users/<UUID>?attributes=groups][Status:403]
        ...

Environment

VMware vRealize Automation 8.1.x

Resolution

Take a Snapshot of the vIDM cluster using vRealize Suite Lifecycle Manager

1. Connect the the vIDM primary database via SSH connection. To validate which is the primary database use this command:

su postgres -c "echo -e 'password'|/opt/vmware/vpostgres/current/bin/psql -h localhost -p 9999 -U pgpool postgres -c \"show pool_nodes\""

2. Extract the database password

cat /usr/local/horizon/conf/db.pwd

3. connect to vPostgres database using the earlier extracted password

/opt/vmware/vpostgres/current/bin/psql -U postgres saas

4. Validate the field "inheritanceAllowed" is set to false

SELECT "inheritanceAllowed" FROM "OAuth2Client" WHERE "clientId"='acs';

5. If the field "inheritanceAllowed" is set to false the update it to true

UPDATE "OAuth2Client" SET "inheritanceAllowed"=true WHERE "clientId"='acs' AND "idOrganization"=2;

6. Exit vPostgres and the shell session on vIDM

\q

7. Open an SSH session to one of the vRA 8.1 nodes and run the deploy script

/opt/scripts/deploy.sh