Usage (Use%) on /common directory reaches 100%

Products

• VMware Telco Cloud Automation

Issue/Introduction

• TCA-M /common/tmp directory full.
• TCA-CP /common directory full
• TCA services have stopped running or  have entered an unresponsive state

Environment

2.3

Cause

• The postgres: autovacuum launcher process on the TCA-CP is unable to run when the /common directory reaches 100% usage. 

Resolution

Addressed in TCA 3.0

Workaround:

Clean up TCA Manager 

• Remove the contents of the /common/tmp folder
  rm -rf /common/tmp
  or
• Reboot the TCA appliance to automatically clear out the /common/tmp directory. 
  Note: Will experience downtime.

Clean up TCA-CP

• SSH to the TCA-CP as the admin user and switch to the root user:
  su
• Stop the TCA services in the following order:
  systemctl stop app-engine
  systemctl stop web-engine
  systemctl stop appliance-management
  systemctl stop kafka
  systemctl stop zookeeper
• Use the following commands to create a /temp directory and move the tca-repository directory:
  mkdir /temp
  mv /common/tca-repository/ /temp
  Note: Do not delete these files as will be utilizing them in a step 14.
• Restart the Postgres service with the following command:
  systemctl restart postgres
• Verify the postgres: autovacuum launcher process is running:
  ps -ef | grep vacuum
• Connect to the Postgres database:
  connect-to-postgres
• Identify the tables with the largest total_database_size by running the following query:
  select current_database() as database,
  pg_size_pretty(total_database_size) as total_database_size,
  schema_name,
  table_name,
  pg_size_pretty(total_table_size) as pretty_total_table_size,
  pg_size_pretty(table_size) as pretty_table_size,
  pg_size_pretty(index_size) as pretty_index_size
  from ( select table_name,
  table_schema as schema_name,
  pg_database_size(current_database()) as total_database_size,
  pg_total_relation_size(quote_ident(table_name)) as total_table_size,
  pg_relation_size(quote_ident(table_name)) as table_size,
  pg_indexes_size(quote_ident(table_name)) as index_size
  from information_schema.tables
  where table_schema=current_schema()
  order by total_table_size
  ) as sizes
  ORDER BY total_table_size desc, table_size desc, index_size desc;
• Run the following command to perform a vacuum on at least the largest three tables using their table_name:
  VACUUM FULL "table_name";
• Exit the Postgres database:
  \q
• Verify the postgres: autovacuum launcher process is running:
  ps -ef | grep vacuum
  Note: If postgres: autovacuum launcher is not displayed, restart the Postgres service:
  systemctl restart postgres
• Restore the tca-repository directory from /temp:
  mv /temp/tca-repository /common
• Restart the TCA services in the following order:
  systemctl start zookeeper
  systemctl start kafka
  systemctl start app-engine
  systemctl start web-engine
  systemctl start appliance-management
• Confirm each TCA service is active (running):
  systemctl status zookeeper
  systemctl status kafka
  systemctl status app-engine
  systemctl status web-engine
  systemctl status appliance-management