vRA 8.4 deployment fails after new vRA load balancer is configured

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• In the deploy.log, you see error similar to:

[2021-04-20 10:44:23.690+0000] Registering embedded vRO

+ set +x
Unable to read vRA certificate configuration for 'lb': Missing certificate configuration for lb
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 228 100 228 0 0 6514 0 --:--:-- --:--:-- --:--:-- 6514
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4278 0 0 100 4278 0 3553 0:00:01 0:00:01 --:--:-- 3553
curl: (22) The requested URL returned error: 400 Bad Request
Failed to register vRO. Will retry in 5 seconds...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4278 0 0 100 4278 0 3854 0:00:01 0:00:01 --:--:-- 3854
curl: (22) The requested URL returned error: 400 Bad Request
Failed to register vRO. Will retry in 15 seconds...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4278 0 0 100 4278 0 3812 0:00:01 0:00:01 --:--:-- 3809
curl: (22) The requested URL returned error: 400 Bad Request
Failed to register vRO. Will retry in 45 seconds...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4278 0 0 100 4278 0 3850 0:00:01 0:00:01 --:--:-- 3850
curl: (22) The requested URL returned error: 400 Bad Request
Maximum number of retries exceeded.
Deployment failed. Collecting log bundle ...

• In the provisioning service logs, you see error similar to:

2021-04-20T10:44:48.551Z WARN provisioning [host='provisioning-service-app-6bd6c5774b-9g754' thread='xn-index-queries-25' user='<USERNAME>' org='<Org_UUID>' trace='<UUID>' parent='' span='<UUID>'] c.v.a.c.e.EndpointCreationTaskService.log:453 - [8282/provisioning/requests/endpoint-creation-tasks/<UUID>] Cloud account name must be unique. Error: com.vmware.xenon.common.LocalizableValidationException: Cloud account name must be unique
at com.vmware.admiral.common.util.ValidationUtils.toLocalizableException(ValidationUtils.java:93)
at >com.vmware.admiral.compute.endpoint.EndpointCreationTaskService.lambda$validateNameUniqueness$53(EndpointCreationTaskService.java:1634)
at java.base/java.util.function.Function.lambda$andThen$1(Function.java:88)
at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at com.vmware.xenon.common.DeferredResult.complete(DeferredResult.java:461)
at >com.vmware.xenon.common.ServiceRequestSender.lambda$sendWithDeferredResult$1(ServiceRequestSender.java:108)
at com.vmware.xenon.common.Operation.lambda$nestCompletion$1(Operation.java:1662)
at com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1631)
at com.vmware.xenon.common.Operation.complete(Operation.java:1458)
…
2021-04-20T10:44:48.560Z WARN provisioning [host='provisioning-service-app-6bd6c5774b-9g754' thread='xn-index-queries-10' user='<USERNAME>' org='<ORG_UUID>' trace='<UUID>' parent='' span='<UUID>'] >c.v.a.c.e.EndpointCreationTaskService.validateStageTransitionAndState:414 - [/provisioning/requests/endpoint-creation->tasks/<UUID>] Referer http://<IP>:8282/provisioning/requests/endpoint-creation->tasks/592cda0e-dab0-42ab-abae-edea057ad721 is patching us to failure during subStage VALIDATE_ENDPOINT_STATE: {
"message": "Cloud account name must be unique",
"statusCode": 400,
"errorCode": 0,
"serverErrorId": "<UUID>",
"documentKind": "com:vmware:xenon:common:ServiceErrorResponse"
}

Note: Run the following command in all of the vRA nodes and search for the the time of the embedded vRO registration from the deploy.log to get the provisioning service logs :
less /services-logs/prelude/provisioning-service-app/file-logs/provisioning-service-app.log

Environment

VMware vRealize Automation 8.4.x

Cause

The embedded vRO registration in vRA 8.4 will fail to update the embedded vRO endpoint if new vRA LB is configured.
The vRO registration code in vRA 8.4 (/opt/scripts/register_vro_endpoint.sh) is modified to search for the previous embedded vRO endpoint, using the following criteria:
single node: host = LB or host = node1
cluster: host = LB or host = node1 or host = node2 or host = node3
Introducing new LB value will cause the above search to fail and will try to create new embedded vRO endpoint, which will fail because one already exists.

Resolution

1. Change the embedded vRO registration code (/opt/scripts/register_vro_endpoint.sh) in vRA 8.4 with the one attached in this KB.
Note: The vRO registration code must be changed in all of the vRA nodes

• Rename the existing vRO registration code with:

mv /opt/scripts/register_vro_endpoint.sh /opt/scripts/register_vro_endpoint.sh.bak

• Copy the new vRO registration code from this KB in the /opt/scripts folder of the vRA appliance
• Edit the file permissions of the new vRO registration code with:

chmod 700 /opt/scripts/register_vro_endpoint.sh

• Check the file permissions with:

ls -la /opt/scripts/register_vro_endpoint.sh
 

The command result should be similar to:
-rwx------ 1 root root 2737 Apr 19 13:28 /opt/scripts/register_vro_endpoint.sh

2. After these steps, run /opt/scripts/deploy.sh again. The vRA development should pass successfully.