Mismatch identified in the status of the volume attached to an instance after a failed resize attempt.
Article ID: 382096
Updated On:
Products
Issue/Introduction
Listing details of the instance using 'openstack server show' command dispalys the volumes attached to the instance and the status of the volumes show as "attached".
However, listing the details of the volume using 'openstack volume show' command, shows the status of the volume as "reserved" and does not list the instances associated with it.
#openstack server show <instance uuid>
+-----------------------------+-------------------------------------------------------------------------------------+ | Field | Value | +-----------------------------+-------------------------------------------------------------------------------------+ | OS-DCF:diskConfig | AUTO | | OS-EXT-AZ:availability_zone | test-AZ | | OS-EXT-STS:power_state | Shutdown | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2024-07-03T04:37:45.000000 | | OS-SRV-USG:terminated_at | None | | created | 2024-07-03T04:37:20Z | | flavor | large | | hostId | ########-####-####-####-############ | | id | ########-####-####-####-############ | | image | image-name (image-uuid) | | key_name | ####-seckey | | name | test-instance | | progress | 67 | | project_id | ########-####-####-####-############ | | properties | | | security_groups | name='default' | | | name='default' | | status | ACTIVE | | updated | 2024-08-09T05:40:33Z | | user_id | ########-####-####-####-############ | | volumes_attached | id='########-####-####-####-###########A' | | | id='########-####-####-####-###########B' | | | id='########-####-####-####-###########C' | +-----------------------------+-------------------------------------------------------------------------------------+
#openstack volume show <volume uuid> +------------------------------+------------------------------------------------------------------+ | Field | Value | +------------------------------+------------------------------------------------------------------+ | attachments | [] | | availability_zone | Test-AZ | | bootable | false | | consistencygroup_id | None | | created_at | 2024-07-09T02:39:21.000000 | | description | | | encrypted | False | | id | ########-####-####-####-############ | | multiattach | False | | name | test-volume-01 | | os-vol-tenant-attr:tenant_id | ########-####-####-####-############ | | properties | | | replication_status | None | | size | 180 | | snapshot_id | None | | source_volid | None | | status | reserved | | type | Test | | updated_at | 2024-07-31T04:25:04.000000 | | user_id | ########-####-####-####-############ | +------------------------------+------------------------------------------------------------------+
Environment
7.x
Cause
This issue occurs after a failed attempt to resize the instance.
Resolution
Validate the status of the volume from Nova & Cinder database by using the below queries and make a note of the "attachment id" and the "id" from the output.
#select * from cinder.volumes where id='########-####-####-####-############';
+---------------------+---------------------+------------+---------+--------------------------------------+--------+------------------------------------------------------------------+----------------------------------+-------------------------------------------------------------------------------+------+-------------------+----------+---------------+---------------------+---------------------+---------------+----------------------------+---------------------+-------------------+---------------+-------------+--------------------------------------+--------------+----------+-------------------+----------+-------------------+------------------+--------------------+-----------------------------+-------------------------+---------------------+-------------+-------------+-----------------+--------------+----------+--------------------------------------+----------------+| created_at | updated_at | deleted_at | deleted | id | ec2_id | user_id | project_id | host | size | availability_zone | status | attach_status | scheduled_at | launched_at | terminated_at | display_name | display_description | provider_location | provider_auth | snapshot_id | volume_type_id | source_volid | bootable | provider_geometry | _name_id | encryption_key_id | migration_status | replication_status | replication_extended_status | replication_driver_data | consistencygroup_id | provider_id | multiattach | previous_status | cluster_name | group_id | service_uuid | shared_targets |+---------------------+---------------------+------------+---------+--------------------------------------+--------+------------------------------------------------------------------+----------------------------------+-------------------------------------------------------------------------------+------+-------------------+----------+---------------+---------------------+---------------------+---------------+----------------------------+---------------------+-------------------+---------------+-------------+--------------------------------------+--------------+----------+-------------------+----------+-------------------+------------------+--------------------+-----------------------------+-------------------------+---------------------+-------------+-------------+-----------------+--------------+----------+--------------------------------------+----------------+| 2024-07-09 02:39:21 | 2024-07-31 04:25:04 | NULL | 0 | ########-####-####-####-############ | NULL | ########-####-####-####-############ | ########-####-####-####-############ | cinder-volume-0@AZ-comp-##:###.###.###VMwareVcVmdkDriver | 180 | AZ-comp-##| reserved | reserved | 2024-07-09 02:39:21 | 2024-07-09 02:39:37 | NULL | volume-01 | | NULL | NULL | NULL | ########-####-####-####-############ | NULL | 0 | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 0 | NULL | NULL | NULL | ########-####-####-####-############ | 0 |
#select * from nova.block_device_mapping where volume_id='########-####-####-####-############';
-----------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+------+--------------------------------------+--------------------------------------+-------------+| 2024-07-09 02:43:00 | 2024-07-31 04:25:04 | NULL | 159042 | /dev/sdb | 0 | NULL | ########-####-####-####-############ | 180 | NULL | {"driver_volume_type": "vmdk", "data": {"volume": "vm-#####", "volume_id": "########-####-####-####-############", "name": "########-####-####-####-############", "profile_id": "########-####-####-####-############", "qos_specs": null, "access_mode": "rw", "encrypted": false}, "status": "reserved", "instance": "########-####-####-####-############", "attached_at": "", "detached_at": "", "volume_id": "########-####-####-####-############", "serial": "########-####-####-####-############"} | ########-####-####-####-############ | 0 | volume | volume | NULL | NULL | NULL | NULL | NULL | NULL | ########-####-####-####-############ | ########-####-####-####-############ | NULL |
#select * from block_device_mapping where volume_id='########-####-####-####-############';
[2024-09-06 11:17:15.081] | 2024-07-09 02:43:00 | 2024-07-09 02:43:12 | 2024-07-31 04:25:04 | 1 | ########-####-####-####-############ | ########-####-####-####-############ | ########.####.############ | ########-####-####-####-############ | /dev/sdb | 2024-07-09 02:43:08 | 2024-07-31 04:25:04 | rw | detached | {"volume": "vm-#####", "volume_id": "########-####-####-####-############", "name": "########-####-####-####-############", "profile_id": "########-####-####-####-############", "qos_specs": null, "access_mode": "rw", "encrypted": false, "driver_volume_type": "vmdk", "attachment_id": "########-####-####-####-############"} | {"ip": "###.###.###", "initiator": null, "host": "###.###.###", "instance": "vm-#####", "mountpoint": "/dev/sdb"} |[2024-09-06 11:17:15.089] | 2024-07-31 04:25:04 | 2024-07-31 04:25:04 | NULL | 0 | ########-####-####-####-############ | ########-####-####-####-############ | NULL | ########-####-####-####-############ | NULL | NULL | NULL | null | reserved | {} | NULL
Make a note of the "attachment id" and the "id" from the above queries.
Run the following SQL queries to correct the database.
update nova.block_device_mapping set attachment_id='########-####-####-####-############' where id=159042;update cinder.volume_attachment set deleted=0, deleted_at=NULL, detach_time=NULL, attach_status='attached' where id='########-####-####-####-############';update cinder.volume_attachment set deleted=1, deleted_at=current_date() where id='########-####-####-####-############';
Additional Information
Please ensure that there is a backup Backup and Restore of the openstack deployment before carrying out the below operation
Feedback
