Control Center access is lost after the configured vCenter Authentication provider certificates expire or change

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• You are attempting to access VMware Aria Automation Orchestrator 8.x (formerly VMware vRealize Orchestrator) services over the URL https://OrchestratorFQDN but a Service Unavailable error is seen.
• Attempting to access https://vROFQDN/vco-controlcenter a Bad Gateway error is seen.
• kubectl get pod -n prelude returns a STATUS of CrachLoopBackOff with a large number of RESTARTS for the vco-app-XXXXXXXXX-xxxxx pod
• kubectl -n prelude delete pod vco-app-XXXXXXXXX-xxxxx does not recover the pod state.
• The VMware vCenter Server appliance certificates have been recently updated which is the Authentication provider for the VMware Aria Automation Orchestrator 8.x instance.

Environment

VMware vRealize Orchestrator 8.x
VMware Aria Automation Orchestrator 8.x

Cause

• This issue is most commonly seen when standalone / clustered VMware Aria Automation Orchestrator instances Authentication provider certificates have expired or have been replaced.

Resolution

VMware is aware of this issue. A fix is being considered for a future release. See the Workaround below for additional information.

Workaround:

Prerequisites

• You have the root user and access to the VMs with SSH.
• You have backups of the VMware Aria Automation Orchestrator 8.x appliance(s).
  • You must backup your VMware Automation Orchestrator VMs at the same time.
  • If you are making the snapshots manually, you must start the snapshots of the second and the third node not more than 40 seconds after you start the snapshots for the first node.

Procedure

1. SSH or PuTTy into one of the nodes within the cluster.
2. Isolate the vco-app pod ID as this value is generated upon pod creation:

   kubectl get pods -n prelude

3. Run the following command to enter a bash shell within the vco-app:

   kubectl -n prelude exec -it vco-app-7fbc9c65cc-2vm25 -c vco-server-app -- bash
   

4. Run the following command to to expose the vro-configure-inner.sh shell script within the pod:

   rpm -hiv --nodeps /vco-cfg-cli.rpm

5. Run the following command to update the aliases vco.vsphere.lookup-service.ssl.certificate and vco.sso.ssl.certificate within the keystore. Be sure to replace vSphere-Auth-Provider-URI with the actual URI for the vSphere Authentication provider in the italicized text:

   /usr/lib/vco-cli/bin/vro-configure-inner.sh trust --alias vco.vsphere.lookup-service.ssl.certificate --uri vSphere-Auth-Provider-URI --accept

Note: If you see the following message in the Control Center logs /services-logs/prelude/vco-app/file-logs/vco-controlcenter-app.log 

Failed to instantiate [com.vmware.vcac.authentication.http.configuration.ConfigurationDataAdapter]: Constructor threw exception; nested exception is java.lang.NullPointerException

Try the following command in replace of Step #5, then complete Step #6-#9:

/usr/lib/vco-cli/bin/vro-configure-inner.sh reset-authentication --enable-legacy-account

6. Type exit followed by the Enter key (carriage return) to exit bash:
7. To stop all services, run:

   /opt/scripts/deploy.sh --shutdown

8. To start all services, run

   /opt/scripts/deploy.sh

9. Validate the deployment has finished by reviewing the output from the deploy.sh script followed by attempting to access the previously inaccessible URLs.