After the reboot, the container 'contour-envoy-#####' is no longer starting

Products

• VCF Automation

Issue/Introduction

After rebooting the Orchestrator appliance, the contour-envoy pod is showing as 1/2 Running 

The Orchestrator UI is not accessible.

In particular, if the appliance has been running for more than a year, the internal gRPC connection certificate expires, making the UI inaccessible.

• The certificate for internal gRPC expires.
  

  kubectl get secret -n prelude contourcert -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates
  notBefore=####
  notAfter=#### <- out of date
  
  kubectl get secret -n prelude envoycert -o jsonpath='{.data.ca\.crt}' | base64 -d| openssl x509 -noout -dates
  notBefore=####
  notAfter=#### <- out of date

  
  
  
• envoy.log
  

  [timestamp][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] StreamListeners gRPC config stream to contour closed since 214s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: TLS_error:|########:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end:TLS_error_end

Environment

Aria Orchestrator Standalone 8.x

Cause

The appliance was rebooted without stopping the deployment services.

 

 

 

Resolution

Restart the deployment services

1. Connect to the console of one node in the cluster or SSH to one node
   
   
2. Stop the services (this will stop services on all nodes in the cluster) by running:
   /opt/scripts/svc-stop.sh
sleep 120
/opt/scripts/deploy.sh --onlyClean
   
   
3. After the Deploy script finishes, and the command prompt reappears, confirm that they have shut down by running:
   kubectl -n prelude get pods
   The output will show "No resources found in prelude namespace."
   
   
4. Verify that the kube-system pods are Running or Completed by running:
   kubectl get pods --all-namespaces
   
   
5. Start the Services by running:
   /opt/scripts/deploy.sh
When it is finished it will report "Prelude has been deployed successfully"
   
   
6. Login into the UI at https://<Your_Orchestrator_FQDN>/orchestration-ui/

Additional Information

When rebooting the appliances follow the steps in Rebooting the Aria Orchestrator Standalone Appliances