Purge Nova database of deleted instances

Products

• VMware Integrated OpenStack

Issue/Introduction

• Upgrade of VMware Integrated Openstack (ie 5.1 to version 6.x or 7.x) takes a long time to upgrade Nova database
• Large, heavily used environments when patching or upgrading regardless of version takes a long time.
• Performance issues when interacting with the Nova service.

Environment

7.x

Cause

In OpenStack systems, Nova performs soft deletions.  Meaning it marks the database entry as deleted.  Depending on usage, cleanup of these records is periodically needed.

Resolution

:
Warning: This procedure modifies the database. Ensure to take a backup of the deployment before proceeding.  See Backup and Restore pg 160.

1. Backup Nova db.

osctl exec -ti mariadb-server-0 -- mysqldump --defaults-file=/etc/mysql/admin_user.cnf -R nova_cell0 > /tmp/nova_cell0_7.2.sql
osctl exec -it mariadb-server-0 -- mysqldump --defaults-file=/etc/mysql/admin_user.cnf -R nova_api > /tmp/nova_api_7.2.sql
osctl exec -ti mariadb-server-0 -- mysqldump --defaults-file=/etc/mysql/admin_user.cnf -R nova > /tmp/nova_7.2.sql

2. Login to nova-api-osapi-xxxxx  pod.

osctl get pods | grep nova-api-osapi
osctl exec -it nova-api-osapi-################ bash

3. Clean up  Nova database.

This will move deleted rows from production tables to shadow tables.  --verbose will print the results of the archive operation for any tables that were changed.

# nova-manage db archive_deleted_rows --verbose --until-complete --max_rows 10000
Archiving....complete
+------------------------------+-------------------------+
| Table                        | Number of Rows Archived |
+------------------------------+-------------------------+
| API_DB.instance_group_member | 0                       |
| API_DB.instance_mappings     | 22                      |
| API_DB.request_specs         | 22                      |
| block_device_mapping         | 22                      |
| instance_actions             | 53                      |
| instance_actions_events      | 53                      |
| instance_extra               | 22                      |
| instance_faults              | 1                       |
| instance_id_mappings         | 22                      |
| instance_info_caches         | 22                      |
| instance_metadata            | 1                       |
| instance_system_metadata     | 256                     |
| instances                    | 22                      |
| virtual_interfaces           | 22                      |
+------------------------------+-------------------------+

This will delete rows from shadow tables.  --verbose will cause information to be printed about purged records.

# nova-manage db purge --all --verbose
DB: Deleted 22 rows from shadow_block_device_mapping based on timestamp column (n/a)
DB: Deleted 53 rows from shadow_instance_actions based on timestamp column (n/a)
DB: Deleted 53 rows from shadow_instance_actions_events based on timestamp column (n/a)
DB: Deleted 22 rows from shadow_instance_extra based on timestamp column (n/a)
DB: Deleted 1 rows from shadow_instance_faults based on timestamp column (n/a)
DB: Deleted 22 rows from shadow_instance_id_mappings based on timestamp column (n/a)
DB: Deleted 22 rows from shadow_instance_info_caches based on timestamp column (n/a)
DB: Deleted 1 rows from shadow_instance_metadata based on timestamp column (n/a)
DB: Deleted 256 rows from shadow_instance_system_metadata based on timestamp column (n/a)
DB: Deleted 22 rows from shadow_instances based on timestamp column (n/a)
DB: Deleted 22 rows from shadow_virtual_interfaces based on timestamp column (n/a)

 Note: The command will take some time to execute especially with large amounts of records.

 

Additional Information

nova-manage command

Cleanup soft deleted volumes

Glance API Performance