Cleanup soft deleted volumes
search cancel

Cleanup soft deleted volumes

book

Article ID: 321773

calendar_today

Updated On:

Products

VMware Integrated OpenStack

Issue/Introduction

Symptoms:

  • Cinder API requests through Horizon or through command line are slow



Environment

7.x

Cause

Like most OpenStack systems, Cinder performs soft deletions when it deletes records from its database. Depending on usage, cleanup of these records is periodically needed.

Resolution

Impact/Risks:
Warning: This procedure modifies the database. Ensure to take a backup of the deployment before proceeding.  See Backup and Restore

  1. Login to the cinder-volume pod
# osctl exec -it cinder-volume-0 bash
  1. Use the cinder-manage tool to purge the soft-deleted rows from all tables
# cinder-manage db purge 60
2023-04-24 05:20:31.109 55 DEBUG oslo_db.sqlalchemy.engines [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python3.7/site-packages/oslo_db/sqlalchemy/engines.py:305
2023-04-24 05:20:31.214 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=volume_glance_metadata
2023-04-24 05:20:31.217 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=snapshot_metadata
2023-04-24 05:20:31.220 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=attachment_specs
2023-04-24 05:20:31.222 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=volume_type_projects
2023-04-24 05:20:31.224 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=volume_type_extra_specs
2023-04-24 05:20:31.227 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=volume_metadata
2023-04-24 05:20:31.229 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=volume_attachment
2023-04-24 05:20:31.231 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=volume_admin_metadata
2023-04-24 05:20:31.233 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=transfers
2023-04-24 05:20:31.236 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=snapshots
2023-04-24 05:20:31.238 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=group_volume_type_mapping
2023-04-24 05:20:31.240 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=workers
2023-04-24 05:20:31.242 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=volumes
2023-04-24 05:20:31.244 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=volume_types
2023-04-24 05:20:31.247 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=reservations
2023-04-24 05:20:31.250 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=group_type_specs
2023-04-24 05:20:31.252 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=group_type_projects
2023-04-24 05:20:31.254 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=group_snapshots
2023-04-24 05:20:31.256 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=cgsnapshots
2023-04-24 05:20:31.259 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=backup_metadata
2023-04-24 05:20:31.261 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=quotas
2023-04-24 05:20:31.263 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=quota_usages
2023-04-24 05:20:31.265 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=quota_classes
2023-04-24 05:20:31.268 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=quality_of_service_specs
2023-04-24 05:20:31.389 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=messages
2023-04-24 05:20:31.392 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=groups
2023-04-24 05:20:31.394 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=group_types
2023-04-24 05:20:31.396 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=encryption
2023-04-24 05:20:31.398 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=consistencygroups
2023-04-24 05:20:31.401 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=clusters
2023-04-24 05:20:31.403 55 INFO cinder.db.sqlalchemy.api [req-77a3c9f0-16d0-48b6-bf73-bc19b7596f40 - - - - -] Purging deleted rows older than age=60 days from table=backups
 
Usage:
  cinder-manage db purge [<number of days>]
    Purge database entries that are marked as deleted, that are older than the number of days specified.

Note:  This command can fail with an error similar to "Cannot delete or update a parent row: a foreign key constraint fails" when trying to purge from the services or volumes tables.  Contact Support for further assistance.  Capture the results from the cinder-manage command and a copy of the cinder database will be needed.

 
 



Additional Information

Glance API Performance
Purge Nova database of deleted instances
Big tables in Openstack database Alarm

To capture the cinder database:

osctl exec -it mariadb-server-0 bash
mysqldump --defaults-file=/etc/mysql/admin_user.cnf --host=localhost  cinder > /tmp/cinder.sql
exit

osctl cp openstack/mariadb-server-0:/tmp/cinder.sql /tmp/cinder.sql

Then you just pull the backup from the manager.  Something like winscp

Powered by Wolken Software