Unable to Delete Stacks Stuck in "DELETE_FAILED" State

Products

• VMware Integrated OpenStack

Issue/Introduction

• Unable to delete stack
• How to forcefully purges this stuck resource
• Stack in DELETE_FAILED state
  $ openstack stack resource list <stack name>
+--------------------+--------------------------------------+-------------------------+-----------------+----------------------+
| resource_name      | physical_resource_id                 | resource_type           | resource_status | updated_time         |
+--------------------+--------------------------------------+-------------------------+-----------------+----------------------+
| <resource name>    | ########-####-####-####-############ | OS::Heat::ResourceGroup | DELETE_FAILED   | <Time Stamp> |    
• Request Timed out    
  $ openstack stack resource <Stack name> <Resource Name>
...
| resource_status_reason | DELETE aborted (Task delete from ResourceGroup "<resource name>" [< physical_resource_id>] Stack "<stack name>" [<stack UUID>] Timed out)   
...

Environment

7.x

Cause

OpenStack fails to delete a heat stack, leaving it in a DELETE_FAILED state, because the request timed out.

Resolution

1. Verify that all the servers in the stack which failed to delete are in a DELETING state.
   #openstack stack list | grep -i DELETE_FAILED | awk '{ print $2 }' | while read i; do openstack stack resource list -n 2 $i | grep -i server | awk '{ print $4}' | while read s; do openstack server show $s -c OS-EXT-STS:task_state -f value ; done; done
   
   
2. Perform a state reset for all the servers in these stacks.
   #openstack stack list | grep -i DELETE_FAILED | awk '{ print $2 }' | while read i; do openstack stack resource list -n 2 $i | grep -i server | awk '{ print $4}' | while read s; do openstack server set --state active $s ; done; done
   
   
3. Verify that there are no more servers in the DELETING state.
   #openstack stack list | grep -i DELETE_FAILED | awk '{ print $2 }' | while read i; do openstack stack resource list -n 2 $i | grep -i server | awk '{ print $4}' | while read s; do STATE=$(openstack server show $s -c OS-EXT-STS:task_state -f value); if [ "$STATE" == "deleting" ]; then echo $n; fi; done; done
   
   
4. Delete the DELETE_FAILED stacks.
   #openstack stack list | grep -i DELETE_FAILED | awk '{ print $2 }' | while read i; do openstack stack delete $i --yes; done