Control Center access is lost after the configured vCenter Authentication provider certificates expire or change
book
Article ID: 326080
calendar_today
Updated On:
Products
VMware Aria Suite
Issue/Introduction
Symptoms:
You are attempting to access VMware Aria Automation Orchestrator 8.x (formerly VMware vRealize Orchestrator) services over the URL https://OrchestratorFQDN but a Service Unavailable error is seen.
Attempting to access https://vROFQDN/vco-controlcenter a Bad Gateway error is seen.
kubectl get pod -n prelude returns a STATUS of CrachLoopBackOff with a large number of RESTARTS for the vco-app-XXXXXXXXX-xxxxx pod
kubectl -n prelude delete pod vco-app-XXXXXXXXX-xxxxx does not recover the pod state.
The VMware vCenter Server appliance certificates have been recently updated which is the Authentication provider for the VMware Aria Automation Orchestrator 8.x instance.
In the vco-controlcenter-app_access.log you see similar 404 entries.
This issue is most commonly seen when standalone / clustered VMware Aria Automation Orchestrator instances Authentication provider certificates have expired or have been replaced.
Resolution
VMware is aware of this issue. A fix is being considered for a future release. See the Workaround below for additional information.
Workaround:
Prerequisites
You have the root user and access to the VMs with SSH.
You have backups of the VMware Aria Automation Orchestrator 8.x appliance(s).
You must backup your VMware Automation Orchestrator VMs at the same time.
If you are making the snapshots manually, you must start the snapshots of the second and the third node not more than 40 seconds after you start the snapshots for the first node.
Procedure
SSH or PuTTy into one of the nodes within the cluster.
Isolate the vco-app pod ID as this value is generated upon pod creation:
kubectl get pods -n prelude
Run the following command to enter a bash shell within the vco-app:
Run the following command to to expose the vro-configure-inner.sh shell script within the pod:
rpm -hiv --nodeps /vco-cfg-cli.rpm
Run the following command to update the aliases vco.vsphere.lookup-service.ssl.certificate and vco.sso.ssl.certificate within the keystore. Be sure to replace vSphere-Auth-Provider-URI with the actual URI for the vSphere Authentication provider in the italicized text:
Type exit followed by the Enter key (carriage return) to exit bash:
To stop all services, run:
/opt/scripts/deploy.sh --shutdown
To start all services, run
/opt/scripts/deploy.sh
Validate the deployment has finished by reviewing the output from the deploy.sh script followed by attempting to access the previously inaccessible URLs.