Cleanup soft deleted volumes
search cancel

Cleanup soft deleted volumes

book

Article ID: 321773

calendar_today

Updated On:

Products

VMware Integrated OpenStack

Issue/Introduction

  • Cinder API requests through Horizon or through command line are slow



Environment

7.x

Cause

In OpenStack systems, Cinder performs soft deletions.  Meaning it marks the database entry as deleted.  Depending on usage, cleanup of these records is periodically needed.

Resolution


Warning: This procedure modifies the database. Ensure to take a backup of the deployment before proceeding.  See Backup and Restore pg 160

  1. Login to the cinder-volume pod
# osctl exec -it cinder-volume-0 bash
  1. Use the cinder-manage tool to purge the soft-deleted rows from all tables

    example results:
    2023-01-01 05:20:31.109 55 DEBUG oslo_db.sqlalchemy.engines [req-########-####-####-####-############ - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python3.7/site-packages/oslo_db/sqlalchemy/engines.py:305
    2023-01-01 05:20:31.214 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=volume_glance_metadata
    2023-01-01 05:20:31.217 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=snapshot_metadata
    2023-01-01 05:20:31.220 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=volume_type_projects
    2023-01-01 05:20:31.224 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=volume_type_extra_specs
    2023-01-01 05:20:31.227 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=volume_metadata
    2023-01-01 05:20:31.229 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=volume_attachment
    2023-01-01 05:20:31.231 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=volume_admin_metadata
    2023-01-01 05:20:31.233 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=transfers
    2023-01-01 05:20:31.236 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=snapshots
    2023-01-01 05:20:31.238 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=group_volume_type_mapping
    2023-01-01 05:20:31.240 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=workers
    2023-01-01 05:20:31.242 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=volumes
    2023-01-01 05:20:31.244 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=volume_types
    2023-01-01 05:20:31.247 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=reservations
    2023-01-01 05:20:31.250 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=group_type_specs
    2023-01-01 05:20:31.252 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=group_type_projects
    2023-01-01 05:20:31.254 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=group_snapshots
    2023-01-01 05:20:31.256 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=cgsnapshots
    2023-01-01 05:20:31.259 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=backup_metadata
    2023-01-01 05:20:31.261 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=quotas
    2023-01-01 05:20:31.263 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=quota_usages
    2023-01-01 05:20:31.265 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=quota_classes
    2023-01-01 05:20:31.268 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=quality_of_service_specs
    2023-01-01 05:20:31.389 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=messages
    2023-01-01 05:20:31.392 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=groups
    2023-01-01 05:20:31.394 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=group_types
    2023-01-01 05:20:31.396 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=encryption
    2023-01-01 05:20:31.398 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=consistencygroups
    2023-01-01 05:20:31.401 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=clusters
    2023-01-01 05:20:31.403 55 INFO cinder.db.sqlalchemy.api [req-########-####-####-####-############ - - - - -] Purging deleted rows older than age=60 days from table=backups
# cinder-manage db purge 60
Usage:
  cinder-manage db purge [<number of days>]
    Purge database entries that are marked as deleted, that are older than the number of days specified.

Note:  This command can fail with an error similar to "Cannot delete or update a parent row: a foreign key constraint fails" when trying to purge from the services or volumes tables.  Contact Broadcom Support for further assistance. 
 
Capture the results from the cinder-manage command.
A copy of the cinder database will be needed.  (see below)

 
 



Additional Information

Glance API Performance
Purge Nova database of deleted instances
Big tables in Openstack database Alarm

To capture the cinder database:

osctl exec -it mariadb-server-0 bash
mysqldump --defaults-file=/etc/mysql/admin_user.cnf --host=localhost  cinder > /tmp/cinder.sql
exit

osctl cp openstack/mariadb-server-0:/tmp/cinder.sql /tmp/cinder.sql

Then you just pull the backup from the manager.  Something like winscp

Powered by Wolken Software