The Git checkout operation fails or takes a long time in Aria Orchestrator (vRO) which causes the vco-app pods to not start

Products

• VCF Automation
• VCF Operations/Automation (formerly VMware Aria Suite)
• VMware vRealize Automation 8.x

Issue/Introduction

• The vco-app pods could not start due to startup probe failed.
• You could receive a startup probe failed error when describing the vco-app pods.
  

  kubectl describe pod -n prelude vco-app

  Startup probe failed: HTTP probe failed with statuscode: 503

• The Aria Orchestrator logs (vco-server-app.log) show an unusually long duration for the checkout operation:

  xx-xx-xxTxx:xx:xx.xxZ DEBUG vco [host='vco-app-id' thread='vcoSystemTaskScheduler-1' user='' org='' trace=''] {} com.vmware.o11n.git.StagingRepositoryService - Checkout took 656662ms
  

Environment

Aria Automation/Orchestrator 8.X

Cause

If the Git checkout takes longer than expected, the vco-app pod will reach its timeout limit and will not start.

Resolution

 CAUTION: After making this change, the "Version History" tab in the Orchestrator client will be empty for all old items. Only new items will have it.

Before applying any changes to the environment, please take a snapshot without memory of the nodes.

1. Take a snapshot of the nodes.
2. SSH into Aria Automation/Orchestrator appliance
3. Create a backup directory for git

   mkdir -p /data/vco/usr/lib/vco/git_backup

4. Move the internal git repository inside the backup directory

   mv /data/vco/usr/lib/vco/app-server/data/git/__SYSTEM.git/ /data/vco/usr/lib/vco/git_backup/

5. Restart the orchestrator pods

   kubectl -n prelude delete pods -lapp=vco-app

6. Monitor if the pod is created successfully 
   

   kubectl get pods -n prelude -o wide -w