Aria Automation pods will not start after executing deploy.sh

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Symptoms:

• Aria Automation is not accessible.
• When checking pods status with command "kubectl -n prelude get pod", some pods' status is "ImageInspectError"
• When checking pod's information with command "kubectl -n prelude describe pod <pod name>", there is "ERROR: readlink /var/lib/docker/overlay2: invalid argument"
• When running script "/opt/scripts/restore_docker_images.sh", it returns "Nothing to load"
• When running script "/opt/scripts/restore_docker_images.sh", it starts loading, but fails with "error creating overlay mount to data/docker/overlay2/#####################/merged: too many levels of symbolic links"
• When checking prelude pods status with command "kubectl -n prelude get pods", may show "No resources found in the prelude namespace"
• When checking kube-system pods status with command "kubectl -n kube-system get pods", pods may be stuck in "ContainerCreating" status

Environment

Aria Automation 8.x

Resolution

Follow procedure below to restore Aria Automation service:

• Run command "/opt/scripts/deploy.sh --shutdown" on any Aria Automation node
• Power off all Aria Automation nodes
• Create snapshot for all Aria Automation nodes
• Power on all Aria Automation nodes
• Run command "kubectl get node" to make sure all nodes are "Ready"
• Run command "docker system prune -af" on all Aria Automation nodes one by one
• Run command "/opt/scripts/restore_docker_images.sh" on all Aria Automation nodes one by one
• Run command "/opt/scripts/deploy.sh" on any Aria Automation node

Additional Information

Additional information can be reviewed for  Aria Orchestrator Automation /var/log is 100% full