Glance API Performance

Products

VMware Integrated OpenStack

Issue/Introduction

Symptoms:

Glance API requests through Horizon or through command line are hanging on:
GET https://[VIO-VIP/FQDN]:9292/v2/images

For example, running the following API command:

#time glance --insecure image-list Produces the following results:

+---------------------------------------------------------------------------------------+-------------------+
| URL | Seconds |
+---------------------------------------------------------------------------------------+-------------------+
| GET https://[VIO-VIP/FQDN]:5000/v3 | 0.03961 |
| POST https://[VIO-VIP/FQDN]:5000/v3/auth/tokens | 0.127194 |
| POST https://[VIO-VIP/FQDN]:5000/v3/auth/tokens | 0.099244 |
| GET https://[VIO-VIP/FQDN]:9292/v2/images | 41.269679 |
| GET https://[VIO-VIP/FQDN]:9292/v2/images?marker=00c32e86-85c6-4059-9e4d-4579dcf1b1d1 | 11.435379 |
| Total | 52.97110599999999 |
+---------------------------------------------------------------------------------------+-------------------+

Note: Soft deleted records could be one of the reasons for performance problems.

Environment

7.x

Cause

Like most OpenStack systems, Glance performs soft deletions when it deletes records from its database (ie marks the record as deleted, not actually deleting the record). Depending on usage, cleanup of these records is periodically needed.

Checking the number of soft deleted records in these tables can be done by running the following:

osctl exec -it mariadb-server-0 bash
mysql --defaults-file=/etc/mysql/admin_user.cnf --host=localhost --connect-timeout 2

select count(*) from task_info;
select count(*) from image_properties where deleted_at !='NULL';
select count(*) from image_tags where deleted_at !='NULL';
select count(*) from image_locations where deleted_at !='NULL';
select count(*) from image_members where deleted_at !='NULL';
select count(*) from tasks where deleted_at !='NULL';

select count(*) from images where deleted_at !='NULL';

Resolution

Impact/Risks:
Warning: This procedure modifies the database. Ensure to take a backup of the deployment before proceeding. See Backup and Restore

Login to the glance-api pod

#osctl get pods | grep glance-api
#osctl exec -it <pod name from above> bash

Use the glance-manage tool to purge the soft-deleted rows from all tables except the images table:

$ glance-manage db purge

>2023-02-03 17:10:57.455 177 DEBUG oslo_db.sqlalchemy.engines [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python3.7/site-packages/oslo_db/sqlalchemy/engines.py:305

>2023-02-03 17:10:57.519 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Purging deleted rows older than 30 day(s) from table task_info

>2023-02-03 17:10:57.564 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Deleted 0 row(s) from table task_info

>2023-02-03 17:10:57.589 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Purging deleted rows older than 30 day(s) from table image_properties

>2023-02-03 17:10:57.606 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Deleted 3 row(s) from table image_properties

>2023-02-03 17:10:57.615 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Purging deleted rows older than 30 day(s) from table image_tags

>2023-02-03 17:10:57.623 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Deleted 0 row(s) from table image_tags

>2023-02-03 17:10:57.641 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Purging deleted rows older than 30 day(s) from table image_locations

>2023-02-03 17:10:57.656 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Deleted 1 row(s) from table image_locations

>2023-02-03 17:10:57.668 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Purging deleted rows older than 30 day(s) from table image_members

>2023-02-03 17:10:57.678 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Deleted 0 row(s) from table image_members

>2023-02-03 17:10:57.678 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Purging deleted rows older than 30 day(s) from table tasks

>2023-02-03 17:10:57.684 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Deleted 0 row(s) from table tasks

Usage: This command takes two optional parameters:

--age_in_days NUM
Only purge rows that have been deleted for longer than NUM days. The default is 30 days.
--max_rows NUM
Purge a maximum of NUM rows from each table. The default is 100.

Use the glance-manage tool to purge the soft-deleted rows from the images table:

Note: Openstack recommends that this is done less frequently than the above tables. See Purging the Images Table

bash-4.4$ glance-manage db purge_images_table

2023-02-03 17:26:43.808 180 DEBUG oslo_db.sqlalchemy.engines [req-0c65abfe-f26b-4456-88c5-964d98c5c0d5 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python3.7/site-packages/oslo_db/sqlalchemy/engines.py:305

2023-02-03 17:26:43.841 180 INFO glance.db.sqlalchemy.api [req-0c65abfe-f26b-4456-88c5-964d98c5c0d5 - - - - -] Purging deleted rows older than 180 day(s) from table images

2023-02-03 17:26:43.850 180 INFO glance.db.sqlalchemy.api [req-0c65abfe-f26b-4456-88c5-964d98c5c0d5 - - - - -] Deleted 0 row(s) from table images

Usage: This command takes two optional parameters:

--age_in_days NUM
Only purge rows that have been deleted for longer than NUM days. The default is 180 days.
--max_rows NUM
Purge a maximum of NUM rows from the images table. The default is 100.

Note: This command can fail with an error similar to "Cannot delete or update a parent row: a foreign key constraint fails" This can happen when you try to purge records from the images table and related records have not been purged from other tables. The purge_images_table command should only be used after the the regular purge command in Step 2 above.

Additional Information

VIO purge Nova database of deleted instances

VIO Cleanup soft deleted volumes

Big tables in Openstack database Alarm