The built-in vRO service in vRA is failing to start due to invalid vco client.

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• The vCO service in VAMI is not coming up.
• The embedded vCO server and Control Center services are failing to start, due to invalid vco client.
• In the vRO logs you see similar to:

[com.vmware.o11n.authentication.services.impl.VRAAuthorizationQueryService]: Constructor threw exception; nested exception is org.springframework.security.authentication.BadCredentialsException: {
  "error": "invalid_client",
  "error_description": "Client is not found."

Environment

VMware vRealize Automation 7.x

Cause

Re-registering the embedded vRO authentication provider from Control Center removes the existing vco client from vIDM and a new one is created and stored in the vRO configuration. However the old vco client remains in the vRA configuration files and might be propagated to vRO later on.

Resolution

Currently, there is no resolution.

Workaround:
To workaround this issue, recreate the vco client in both vRA and vRO configuration files.

1. Backup all vRealize Automation appliance nodes, IaaS nodes and the MSSQL IaaS database.
2. On the master vRA node, delete the line with the original vco client in /etc/vcac/solution-users.properties

For example:
vco=vco-8f2-kob_2w

This will force new vco client creation later on.

3. Stop the vco-services on all vRA nodes: service vco-server stop && service vco-configurator stop
4. Delete the vco registration id on all vRA nodes: rm /etc/vco/app-server/vco-registration-id
5. On the master vRA node, delete all vco VAMI service registrations: vcac-config service-delete --service-name vco
6. On the master vRA node, reconfigure the vRO Control Center authentication: /var/lib/vco/tools/configuration-cli/bin/vro-configure.sh reset-authentication
7. On the master vRA node, reconfigure the vRO Server service: vcac-vami vco-service-reconfigure
8. Wait until the vRO server is fully started.
9. If some of the vRA services on the vRA master node are marked as NOT AVAILABLE, run: service vcac-server restart
10. Once all VAMI services on the master vRA node are marked as REGISTERED, re-join the replica nodes to the cluster again - from the replica node VAMI > Cluster page.
11. Login to the vRO Control Center and navigate to the Configure Authentication Provider page. Configure the vRO authentication settings: tenant, AD group, etc. 
12. Start the Control Center service on all replica vRA nodes. 
13. Validate the vRO configuration.