Error: "Service Unavailable" while accessing Aria Automation Orchestrator

Products

VCF Operations/Automation (formerly VMware Aria Suite) VMware Telco Cloud Platform

Issue/Introduction

Symptoms:

You are attempting to access VMware Aria Automation Orchestrator 8.x (formerly VMware vRealize Orchestrator) services over the URL https://OrchestratorFQDN but a Service Unavailable error is seen.
Attempting to access https://vROFQDN/vco a Bad Gateway error is seen.
kubectl get pod -n prelude returns a STATUS of CrachLoopBackOff with a large number of RESTARTS for the vco-app-XXXXXXXXX-xxxxx pod
kubectl -n prelude delete pod vco-app-XXXXXXXXX-xxxxx does not recover the pod state.
The VMware vCenter Server appliance certificates have been recently updated which is the Authentication provider for the VMware Aria Automation Orchestrator 8.x instance.
In the vco-controlcenter-app_access.log you see similar 404 entries.
GET /vco-controlcenter/ HTTP/1.1 404 431 38 ms
vracli service status returns vco Degraded:

Environment

VCF Operation/Automation formerly VMware Aria Automation Orchestrator formerly VMware vRealize Orchestrator 8.9.1, 8.x

TCP: 3.0

Cause

This issue is most commonly seen when standalone / clustered VMware Aria Automation Orchestrator instances Authentication provider certificates have expired or have been replaced.

Resolution

Prerequisites

You have the root user and access to the VMs with SSH.
You have backups of the VMware Aria Automation Orchestrator 8.x appliance(s).
- You must backup your VMware Automation Orchestrator VMs at the same time.
- If you are making the snapshots manually, you must start the snapshots of the second and the third node not more than 40 seconds after you start the snapshots for the first node.

Procedure

SSH or PuTTy into one of the nodes within the cluster.
Isolate the vco-app pod ID as this value is generated upon pod creation:
```
kubectl get pods -n prelude
```
Run the following command to enter a bash shell within the vco-app:
```
kubectl -n prelude exec -it vco-app-7fbc9c65cc-2vm25 -c vco-server-app -- bash
```
Note: If this command returns a message stating that no container with name vco-server-app exists, follow Steps 7 and 8, to redeploy them and then follow procedure from Step 1
Run the following command to to expose the vro-configure-inner.sh shell script within the pod:
```
rpm -hiv --nodeps /vco-cfg-cli.rpm
```
Run the following command to update the aliases vco.vsphere.lookup-service.ssl.certificate and vco.sso.ssl.certificate within the keystore. Be sure to replace vSphere-Auth-Provider-URI with the actual URI for the vSphere Authentication provider in the italicized text:
```
/usr/lib/vco-cli/bin/vro-configure-inner.sh trust --alias vco.vsphere.lookup-service.ssl.certificate --uri vSphere-Auth-Provider-URI --accept
```

Note: If you see the following message in the Control Center logs /services-logs/prelude/vco-app/file-logs/vco-controlcenter-app.log

Failed to instantiate [com.vmware.vcac.authentication.http.configuration.ConfigurationDataAdapter]: Constructor threw exception; nested exception is java.lang.NullPointerException

Try the following command in replace of Step #5, then complete Step #6-#9:

/usr/lib/vco-cli/bin/vro-configure-inner.sh reset-authentication --enable-legacy-account

Type exit followed by the Enter key (carriage return) to exit bash:

To stop all services, run:

/opt/scripts/deploy.sh --shutdown

(For standalone Orchestrator servers, run /opt/scripts/deploy.sh --onlyClean)

To start all services, run
```
/opt/scripts/deploy.sh
```
Validate the deployment has finished by reviewing the output from the deploy.sh script followed by attempting to access the previously inaccessible URLs.