vRealize Automation or vRealize Orchestrator pods fail to start with status ImageInspectError

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• vRA / vRO is not accessible.
• deploy.sh fails to start services
• Running command: kubectl describe pod <name-of-pod>, returns with message Error from server (NotFound): pods <name-of-pod> not found.
• When running command: kubectl -n prelude get pods, the pod status shows ImageInspectError.

root@ [ ~ ]# kubectl -n prelude get pods
NAME                                           READY   STATUS                   RESTARTS   AGE
abx-service-app-6c6d6fdc8-m498x                0/2     Init:0/1                 0          20h
adapter-host-service-app-6f4c48f7c9-48gfh      0/1     Init:0/1                 0          20h
approval-service-app-6c8f5fd877-rh589          0/1     Init:0/2                 0          20h
assessment-service-app-7c9857c779-q49c5        0/1     Init:0/2                 0          20h
catalog-service-app-6d9cdc77db-8qvgs           0/1     Init:0/2                 0          20h
cgs-service-app-77f69d6d7-7sgtg                0/1     Init:0/2                 0          20h
cmx-service-app-7b548c6976-qff5w               0/1     Init:0/1                 0          20h
codestream-app-597ddb5bbd-m87hc                0/1     Init:0/2                 0          20h
ebs-app-7dbdc87c7-9hf4j                        0/1     Init:ImageInspectError   0          20h
form-service-app-5fb589d8f9-4tgnz              0/1     Init:0/1                 0          20h
hcmp-service-app-7bb795cd5-z978s               0/1     Init:0/1                 0          20h
identity-service-app-5f55579cc7-7k7q2          1/1     Running                  0          20h
migration-service-app-6db499898f-4bvdb         0/1     Init:0/1                 0          20h
no-license-app-6f6f58bcc7-9dz7p                1/1     Running                  0          20h
postgres-0                                     1/1     Running                  0          20h
project-service-app-5bb4f6d5cb-j5b76           0/1     Init:0/1                 0          20h
provisioning-service-app-c976b74f8-vzlp7       0/1     Init:0/1                 0          20h
proxy-service-7564886dd-jzrmm                  1/1     Running                  0          20h
rabbitmq-ha-0                                  1/1     Running                  0          20h
relocation-service-app-656ff44c7b-dp6kv        0/1     Init:0/1                 0          20h
tango-blueprint-service-app-7f4768bf48-tt5tg   0/1     Init:0/2                 0          20h
tango-vro-gateway-app-59ccb79bd5-8lnvv         0/1     Init:0/1                 0          20h
terraform-service-app-58ff6b4b5-2b28b          0/2     Init:ImageInspectError   0          20h
user-profile-service-app-785cbc9875-jgqn9      1/1     Running                  0          20h
vco-app-95cd5b9ff-pdfwx                        2/3     ImageInspectError        0          20h

Environment

VMware vRealize Automation 8.x

Resolution

VMware is aware of this issue.

See the Workaround below for additional information and next steps.

Workaround:

Prerequisites

• Please take simultaneous non-memory snapshots of each virtual appliance(s) in the cluster.
• You have access to root user and password
• You have SSH or console access to each virtual appliance.

Procedure

1. SSH / PuTTy into the virtual appliance in the cluster experiencing these Symptoms.
2. Run the following commands
   1. Shutdown services

      /opt/scripts/deploy.sh --shutdown

   2. Remove cleanup files. Perform this on any node experiencing the Symptoms described.

      rm -f /var/vmware/prelude/docker/last-cleanup

   3. Reboot the appliance
   4. Run first-boot checks

      vracli status first-boot -w 1800

   5. Wait or repeat the above until it reports the following

      First boot complete

   6. Start services

      /opt/scripts/deploy.sh

   7. Verify services are available by logging into the vRA / vRO portals and/or ensuring deploy.log reports back Deployment Success.

Additional Information

Impact/Risks:

• vRA UI shows 404 page not found. 
• Services fail to start.
• Deploy.sh fails with an error starting pods.