Openstack volume stuck in deleting status

Products

• VMware Integrated OpenStack

Issue/Introduction

• The delete command was executed against the target volume but the status remained as "deleting" and the deletion remianed unsuccessful.
• There is no deletion task noticed in the vCenter  task and events pane.

#openstack volume delete  <volume id>

Environment

VMware Integrated Openstack 7.x

Cause

The volume may be stuck in  "deleting" state due to various reasons : 

• Backend or database inconsistencies
• Volume still attached on the instance.
• Backend storage issue
• Volume already deleted from backend manually
• Stale database state

Resolution

When the volume is stuck in deleting state it will not allow to perform any further deletion attempts.We need to reset the state and then attempt to delete. 

Steps to delete : 

1. Check if the volume is still attached to the instance, if yes you will need to detach it first and than attempt to delete.

2. Try to make the volume to available state manually and try deleting    

# cinder reset-state --state "available" <volume id>

3. If the above doesn't work, try to reset the state to "Error" and make it to "available" again, than perform the deletion 

  # openstack volume set --state "error"  <volume id> 

  # cinder reset-state --state "available"  <volume id>

4. If the above still fails, you can try to force delete    

#openstack volume delete  --force  <volume id>

After performing the above steps from 1 to 4, volume still remained in "deleting"  status and could not be deleted,we attempted the following

1.  Return the status of the volume to “available” 
2.  Perform migration between Availability zone (AZs)
3.  After the migration is completed, rewrite the target volume information to the correct AZ in VIO.
4.  Execute the delete command.
5. Confirm that the deletion has been completed.

Impact/Risks:

Warning: This procedure modifies the database. Ensure to take a backup of the deployment before proceeding.  See Backup and Restore.

If the deletion is not successful after performing the above steps then mark the volume as deleted at the database level as a last resort. 

Note: The following modification is irreversible. Do NOT perform this unless you are certain that the volume is no longer needed. It is strongly recommended to raise a support request to perform these kind of medications. 

1. Login to DB (e.g., MySQL or MariaDB)

mysql -u root -p
USE cinder;

2.  Find the volume :

#SELECT id, status FROM volumes WHERE id='<volume-id>';

3. Mark it as deleted :

#UPDATE volumes SET deleted=1, status='deleted', deleted_at=NOW() WHERE id='<volume-id>';