Usage on /common directory reaches 100%

Products

• VMware Telco Cloud Automation

Issue/Introduction

This article will guide you through cleaning up the /common directory on the TCA-CP appliance.

TCA services will stop running or enter an unresponsive state if the /common directory reaches 100% usage (Use%).

Environment

2.3

Cause

The postgres: autovacuum launcher process on the TCA-CP is unable to run when the /common directory reaches 100% usage. 

Resolution

Addressed in TCA 3.0

Workaround:

1. SSH to the TCA-CP as the admin user and switch to the root user:
   su
2. Stop the TCA services in the following order:
   systemctl stop app-engine
   systemctl stop web-engine
   systemctl stop appliance-management
   systemctl stop kafka
   systemctl stop zookeeper
3. Use the following commands to create a /temp directory and move the tca-repository directory:
   mkdir /temp
   mv /common/tca-repository/ /temp
   Note: Do not delete these files as will be utilizing them in a step 14.
4. Restart the Postgres service with the following command:
   systemctl restart postgres
5. Verify the postgres: autovacuum launcher process is running:
   ps -ef | grep vacuum
6. Connect to the Postgres database:
   connect-to-postgres
7. Identify the tables with the largest total_database_size by running the following query:
   select current_database() as database,
   pg_size_pretty(total_database_size) as total_database_size,
   schema_name,
   table_name,
   pg_size_pretty(total_table_size) as pretty_total_table_size,
   pg_size_pretty(table_size) as pretty_table_size,
   pg_size_pretty(index_size) as pretty_index_size
   from ( select table_name,
   table_schema as schema_name,
   pg_database_size(current_database()) as total_database_size,
   pg_total_relation_size(quote_ident(table_name)) as total_table_size,
   pg_relation_size(quote_ident(table_name)) as table_size,
   pg_indexes_size(quote_ident(table_name)) as index_size
   from information_schema.tables
   where table_schema=current_schema()
   order by total_table_size
   ) as sizes
   ORDER BY total_table_size desc, table_size desc, index_size desc;
8. Run the following command to perform a vacuum on at least the largest three tables using their table_name:
   VACUUM FULL "table_name";
9. Exit the Postgres database:
   \q
10. Verify the postgres: autovacuum launcher process is running:
    ps -ef | grep vacuum
    Note: If postgres: autovacuum launcher is not displayed, restart the Postgres service:
    systemctl restart postgres
11. Restore the tca-repository directory from /temp:
    mv /temp/tca-repository /common
12. Restart the TCA services in the following order:
    systemctl start zookeeper
    systemctl start kafka
    systemctl start app-engine
    systemctl start web-engine
    systemctl start appliance-management
13. Confirm each TCA service is active (running):
    systemctl status zookeeper
    systemctl status kafka
    systemctl status app-engine
    systemctl status web-engine
    systemctl status appliance-management