Deployment of Aria Automation 8.x fails and deletion of failed deployment is unsuccessful

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

This article will guide users in resolving issues when deleting a failed Aria Automation 8.x deployment stuck in a pending state and provide troubleshooting tips encountered during cancellation attempts.

Symptoms:

After initiating the deletion of the failed deployment, we find it stuck in a pending state for a longer period of time.
Attempts to cancel the deletion action result in another error:

'Error: Internal Server Error [Error Reference ID: ########-######-######-#######-#####]'
Deployment are stuck in Pending Approval state even though it has been approved.

Environment

VMware Aria Automation 8.x

Cause

A Day 2 request not finishing for the deployment may cause no other Day 2 request to start until the previous request is finished.

Resolution

VMware is aware of this issue. See the Workaround below for additional information.

Workaround:

Prerequisites

You have backups of the VMware Aria Automation 8.x appliance(s).
You must back up all VMware Aria Automation appliances, at the same time - simultaneously for all nodes.
If you are taking the snapshots manually, you must start the snapshots of the second and the third node no more than 40 seconds after you start the snapshot for the first node.
When you back up the VMware Aria Automation appliance, disable in-memory snapshots and enable quiescing for versions 8.9 and above.

Procedure

SSH into one of the Aria Automation appliance nodes in your cluster.
Run the following command to open the catalog database:
1. ```
vracli dev psql catalog-db
```
2. Use the following command to adjust the views of the following queries to make them easier to read
  1. ```
  \x auto
```
Find the problematic deployment and select it from the deployment list, verify this is the deployment to fix by running the following query and matching this ID to the one found in the
1. You can locate the the deployment id in the url after the "%2F" (example: https://<Aria Automation URL>/automation/#/service/automation-ui/deployment-ui;ash=%2Fworkload%2Fdeployment%2F########-####-####-####-############)
2. ```
select * from dep_deployment where id = 'deployment_id';
```
If current day 2 request is stuck for longer than the expected runtime, or many days it is necessary to reset the status to allow for new actions to be submitted:
```
select id, name, status from dep_request where deployment_id = 'deployment_id' order by created_at desc;
```

Example:

id                  |     name     |   status  
--------------------------------------+--------------+------------ 
########-######-######-#######-##### | Expire Event | FAILED 
########-######-######-#######-##### | Expire       | SUCCESSFUL 
########-######-######-#######-##### | Create       | FAILED
########-######-######-#######-##### | Delete       | IN-PROGRESS
(4 rows)

Set the deployment request value to FAILED.

Use the deployment id from step 3 and the ID from step 4 for the following step

update dep_request set status = 'FAILED' where id = 'step 4 ########-######-######-#######-#####' and deployment_id = 'step 3 deployment_id';

Login to the Aria Automation user interface and reattempt the deletion of the deployment.

Additional Information

Impact/Risks:

Users may face disruptions in deleting a failed Aria Automation 8.x deployment due to pending Day 2 requests.
The API to delete the failed deployment fails with the below response:
```
"Another conflicting request is already in progress".
```