After the reboot, Aria Automation or Standalone Orchestrator UI fails to load

Products

• VCF Automation

Issue/Introduction

• After rebooting the appliance, the contour-envoy pod is showing as 1/2 Running 
• In particular, if the appliance has been running for more than a year, the internal gRPC connection certificate expires, making the UI inaccessible.

• The certificate for internal gRPC expired.
  

  kubectl get secret -n prelude contourcert -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates
  notBefore=####
  notAfter=#### <- out of date
  
  kubectl get secret -n prelude envoycert -o jsonpath='{.data.ca\.crt}' | base64 -d| openssl x509 -noout -dates
  notBefore=####
  notAfter=#### <- out of date

  
  
  
• Envoy logs located at /var/log/services-logs/prelude/envoy/file-logs/envoy.log 
  

  [timestamp][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] StreamListeners gRPC config stream to contour closed since 214s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: TLS_error:|########:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end:TLS_error_end

Environment

Aria Orchestrator Standalone 8.x

Aria Automation 8.x

Cause

The appliance was rebooted without stopping the deployment services.

Resolution

Restart the deployment services

1. Connect to the console of one node in the cluster or SSH to one node
   
   
2. Stop the services (this will stop services on all nodes in the cluster) by running:
   /opt/scripts/svc-stop.sh
sleep 120
/opt/scripts/deploy.sh --onlyClean
   
   
3. After the Deploy script finishes, and the command prompt reappears, confirm that they have shut down by running:
   kubectl -n prelude get pods
   The output will show "No resources found in prelude namespace."
   
   
4. Verify that the kube-system pods are Running or Completed by running:
   kubectl get pods --all-namespaces
   
   
5. Start the Services by running:
   /opt/scripts/deploy.sh
When it is finished it will report "Prelude has been deployed successfully"
   
   
6. Login into the UI at https://<Your_Orchestrator or Automation_FQDN>/orchestration-ui/

Additional Information

When rebooting the appliances follow the steps in Rebooting the Aria Orchestrator Standalone Appliances