vRealize Orchestrator three-node cluster is in an inconsistent state

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• VMware vRealize Orchestrator validate configuration details:

Orchestrator Cluster in an inconsistent state.

• Cluster tab shows one or more nodes in an error state:

Failed to retrieve the service's health status.

Cannot connect to the remote Control Center by any of the provided endPoints: [EndPoint [hostname=vco-hostname, port=443, useSsl=true], EndPoint [hostname=vco-hostname, port=8282, useSsl=false], EndPoint [hostname=vco-hostname, port=8283, useSsl=true], EndPoint [hostname=10.XXX.XX.XX, port=443, useSsl=true], EndPoint [hostname=10.XXX.XX.XX, port=8282, useSsl=false], EndPoint [hostname=10.XXX.XX.XX, port=8283, useSsl=true]]

• Listing pods displays only one vco-app pod:

Note: Three-node cluster should display 3 vco-app pods.

kubectl -n prelude get pods
NAME                                    READY   STATUS    RESTARTS   AGE
orchestration-ui-app-798964dd7d-h7dp7   1/1     Running   0          28m
orchestration-ui-app-798964dd7d-lcxbj   1/1     Running   0          28m
orchestration-ui-app-798964dd7d-rptlh   1/1     Running   0          28m
postgres-0                              1/1     Running   0          29m
postgres-1                              1/1     Running   0          29m
postgres-2                              1/1     Running   0          29m
proxy-service-64b67b866c-kms2z          1/1     Running   0          29m
proxy-service-64b67b866c-s5z5z          1/1     Running   0          29m
proxy-service-64b67b866c-wh4js          1/1     Running   0          29m
vco-app-5496bf55d-dr6dc                 3/3     Running   0          28m

Environment

VMware vRealize Orchestrator 8.x

Resolution

To resolve the issue, stop and restart all pods by running the command below:

/opt/scripts/svc-stop.sh; sleep 120; /opt/scripts/deploy.sh --onlyClean; sleep 60; /opt/scripts/deploy.sh

All three vco-app pods should now be deployed and in Running state.