Deployment of Aria Automation 8.x fails and deletion of failed deployment is unsuccessful

Products

VMware Aria Suite

Issue/Introduction

This article will guide users in resolving issues when deleting a failed Aria Automation 8.x deployment stuck in a pending state and provide troubleshooting tips encountered during cancellation attempts.

Symptoms:

After initiating the deletion of the failed deployment, we find it stuck in a pending state for a longer period of time.

Attempts to cancel the deletion action result in another error:

'Error: Internal Server Error [Error Reference ID: xxxxxxxxxxx-xxxxxx-xxxxxxx-xxxx]'.

Environment

VMware Aria Automation 8.x

Cause

A Day 2 request not finishing for the deployment may cause no other Day 2 request to start until the previous request is finished.

Resolution

VMware is aware of this issue. See the Workaround below for additional information.

Workaround:

Prerequisites

You have backups of the VMware Aria Automation 8.x appliance(s).
You must back up all VMware Aria Automation appliances, at the same time - simultaneously for all nodes.
If you are taking the snapshots manually, you must start the snapshots of the second and the third node no more than 40 seconds after you start the snapshot for the first node.
When you back up the VMware Aria Automation appliance, disable in-memory snapshots and enable quiescing for versions 8.9 and above.

Procedure

SSH into one of the Aria Automation appliance nodes in your cluster.
Run the following command to open the catalog database:
```
vracli dev psql catalog-db
```
Find the problematic deployment, verify this is the deployment to fix by running the following query and matching this ID to the one found in the
```
select * from dep_deployment where id = 'deployment_id';
```
If current day 2 request is stuck for longer than the expected runtime, or many days it is necessary to reset the status to allow for new actions to be submitted:
```
select id, name, status from dep_request where deployment_id = 'deployment_id' order by created_at desc;
```

Example:

id                  |     name     |   status  
--------------------------------------+--------------+------------ 
xxxxxxxx-xxxxxx-xxxxxx-xxxxxxx-xxxxx | Expire Event | FAILED 
xxxxxxxx-xxxxxx-xxxxxx-xxxxxxx-xxxxx | Expire       | SUCCESSFUL 
xxxxxxxx-xxxxxx-xxxxxx-xxxxxxx-xxxxx | Create       | FAILED
xxxxxxxx-xxxxxx-xxxxxx-xxxxxxx-xxxxx | Delete       | IN-PROGRESS
(4 rows)

Set the deployment request value to FAILED.

update dep_request set status = 'FAILED' where id = 'xxxxxxxx-xxxxxx-xxxxxx-xxxxxxx-xxxxx' and deployment_id = 'deployment_id';

Login to the Aria Automation user interface and reattempt the deletion of the deployment.

Additional Information

Impact/Risks:

Users may face disruptions in deleting a failed Aria Automation 8.x deployment due to pending Day 2 requests.
The API to delete the failed deployment fails with the below response:
```
"Another conflicting request is already in progress".
```