AWS MariaDB 10.x upgrade procedure for BOSH Director
search cancel

AWS MariaDB 10.x upgrade procedure for BOSH Director

book

Article ID: 293824

calendar_today

Updated On: 08-06-2024

Products

Operations Manager

Issue/Introduction

Introduction

The BOSH director is not compatible with newer versions of MariaDB, including 10.5. Additionally, Tanzu Application Service (TAS) has some compatibility issues with MariaDB 10.2+, so we are requiring customers to move to MySQL for the long-term. 

For the procedure to migrate Tanzu Application Service (TAS), please refer to the following article: AWS MariaDB 10.1 to MySQL 5.7 “downtime” upgrade instructions using mysqldump.


This procedure was tested using mysql and mysqldump clients packaged on Ops Manager 2.10.7. For the best experience, use the following versions of these clients:
 

  • mysql  Ver 15.1 Distrib 10.4.17-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
  • mysqldump  Ver 10.18 Distrib 10.4.17-MariaDB, for debian-linux-gnu (x86_64)

Prerequisites

  • Ops Manager using external MySQL-compatible DB for the BOSH Director.
  • Existing RDS database instance running MariaDB 10.0, 10.1, 10.2, 10.3, or 10.4 (does not apply to 10.5)
  • The ability to create a new RDS database instance running MySQL 5.7.
  • A jumpbox with at least 50 GB free disk space, a mysql client, and a mysqldump.

Impact during the migration

This is a low impact migration. Processes that use the BOSH Director, such as HealthWatch and On-Demand-Brokered Services, will not be able to perform BOSH deployments until the process is complete. Since BOSH is not a runtime dependency of TAS itself, there will be no platform impact. The resurrector will not work until the process is complete. 

To summarize, during the migration the following processes won’t be working:

  • Platform redeploys / configuration changes
  • Processes that use the BOSH Director such as:
    • HealthWatch
    • On-Demand-Brokered Services
  • The BOSH resurrector



Environment

Product Version: 2.10

Resolution

Procedure

1. Make sure you have the latest version of om installed from https://github.com/pivotal-cf/om/releases.

Note: If you'd prefer to use Ops Manager's API directly, the referenced om commands have simple API alternatives.

2. Determine where the BOSH Director database is currently located:

GUI:
  • Log in to Ops manager
  • Go to BOSH Director tile
  • Click "Director Config"
  • At the bottom of the page, capture the value of the "Database" field.
CLI:

Run the following command and capture the value / output:
om staged-director-config | bosh interpolate - --path /properties-configuration/director_configuration/external_database_options/database

3. Shutdown the BOSH Director processes so that no write operations take place during migration:
a. Check for no active deploys
bosh tasks
b. SSH into the BOSH Director.
c. Become root
sudo su -
d. Stop all processes
monit stop all
e. wait for all monit jobs to stop
f.  gracefully shutdown bosh director using IAAS console or following command
sudo shutdown now
g. Monitor the VM either in IaaS (until it is in a powered off state) or until the VM stops responding to ping.


4. SSH to Ops Manager VM where you can find the mysql and mysqldump clients.

To validate row count for your tables, run the following command:

~$ mysql -h <MY TARGET DB HOSTNAME> -u <MY SOURCE DB ADMIN USERNAME> -p -e "SELECT table_name, table_rows FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = '<DATABASE NAME>';"


Note: You can run this command at the end of the migration to confirm it was successful. The counts might be slightly different because of how the counts are approximated. To get a precise count for a table t in the database db, then run the command: SELECT COUNT(*) FROM db.t; 


5. Dump the source MariaDB database:

mysqldump \
--single-transaction \
-h <MY SOURCE DB HOSTNAME> \
-u <MY SOURCE DB ADMIN USERNAME> \
-p \
--databases <DB NAME FROM PREVIOUS STEP> > /tmp/DirectorDB_dump.sql

6. Import the schema into the new target MySQL instance:
mysql \
-h <MY TARGET DB HOSTNAME> \
-u <MY TARGET DB ADMIN USERNAME> \
-p  \
<DB NAME FROM PREVIOUS STEP> < /tmp/DirectorDB_dump.sql

7. Put the Ops Manager into Advanced Mode (CLI only). This allows you to edit the database host information:
om curl -s -p /api/v0/staged/infrastructure/locked -x PUT --data '{"locked" : "false"}'

8. Update the configuration using either GUI or CLI method.

GUI:
  • Log in to Ops Manager
  • Go to BOSH Director tile
  • Click "Director Config"
  • Change the "Hostname" field to the new Database Hostname
  • Click "Save"
CLI:

Create a .yaml file called update-hostname.yml that looks like this:
properties-configuration:
  director_configuration:
    external_database_options:
      host: <MY TARGET DB HOSTNAME>

Use it by running:
 om configure-director -c update-hostname.yml 

9. Apply Changes only for the BOSH Director product. Operations Manager will recreate the bosh director using the new mysql hostname.

WARNING: Once this step has been successfully completed, and the BOSH director is now pointing at the new database, you can no longer rollback because the old BOSH database is very quickly going out of sync with the IaaS.

GUI:
  • From the Installation Dashboard, click "Review Pending Changes"
  • Uncheck "Select All Products" and ensure that the BOSH Director tile is the only checkbox selected (it cannot be unselected).
  • Click "Apply Changes"
CLI:

Run the following command:
om apply-changes --skip-deploy-products 

10. Take the Ops Manager out of Advanced Mode (CLI only):
om curl -s -p /api/v0/staged/infrastructure/locked -x PUT --data '{"locked" : "true"}

Note: This should automatically occur after a successful Apply Changes.