Migrating BOSH Director of the Local Blobstore to External Blobstore

Products

• Operations Manager

Issue/Introduction

Eventually the BOSH Director local blobstore can become large and difficult to maintain. A common challenge is BOSH Backup and Restore (BBR) backups failing because of space issues on the director persistent disk - scaling up the director VM instance storage may not be an option for some environments.

This article will describe the recommended method for migrating the BOSH Director of the local blobstore. Please be sure to take a backup using the standard BOSH Backup and Restore procedures before attempting this process.

Resolution

Preparation

Before starting the migration, it is important to ensure you download all the versions of the installed tiles to a jump box or the Tanzu Operations Manager VM. When importing a tile, Operations Manager will unpack all the releases of that tile into the directory "/var/tempest/releases/". However, when you upgrade Operations Manager, it does not carry over any releases that were previously imported. Downloading the tiles from Tanzu Network ahead of time ensures that you will have all the binaries needed to fully restore the contents of the directors' local blobstore to the new external blobstore.

All tiles will have the .pivotal extension when downloaded from the Tanzu network. Despite the given extension, they will either be a .zip or .tgz file. So you can rename them and unpack them using unzip or tar cli commands.

Each tile will have a folder <tile archive>/releases/ that includes all of the releases for the given tile. All the release tarball files will be needed to restore the Directors' blobstore.

Procedure

1. Make sure to take a backup of the BOSH Director using BBR before starting this procedure.

2. By default, Operations Manager will not allow you to modify the Director blobstore setting in Director Tile -> Director Config. You must first enable Advanced Mode in Operations Manager.

3. While in advanced mode, make the changes to the BOSH director blobstore and save. 

4. Disable Advanced Mode in Operations Manager.

5. Apply Changes to the BOSH Director only.
 

• Note: At this point, you will not be able to Apply Changes to installed tiles until the following steps are completed.

6. After the Apply Changes are successful, upload any releases that exist in /var/tempest/releases and /var/tempest/internal_releases directory which reside on the Operations Manager VM.

Scripted method

ls /var/tempest/releases/ | while read line ; do bosh upload-release /var/tempest/releases/$line --fix; done

ls /var/tempest/internal_releases/ | while read line ; do bosh upload-release /var/tempest/internal_releases/$line --fix; done

Simple Command Method

bosh upload-release <release-file> --fix

1. Also review the contents of /var/tempest/releases and upload all the tile releases that are missing from the Operations Manager VM. The procedure to do this is discussed in the Preparation steps above.

2. Once all the releases have been re-uploaded to the BOSH Director, we then need to configure the Director to recreate all VMs. The reason why we need to recreate all VMs is because the bosh agents on the instance VM's will have a reference to the old blobstore configuration. We need to recreate the VM's to ensure the BOSH agents learn about the new blobstore.  

• Go to Director Tile -> Director Config and enable "Recreate VMs deployed by the BOSH Director"
• TIP: You can verify which blobstore the BOSH agent is configured to use by reading file "/var/vcap/bosh/etc/blobstore-dav.json" on the instance VM.

3. If you have any on-demand service tiles then please enable the recreate errands for those tiles. 

4. In order to consider this procedure successful, all VM's in the BOSH environment must be recreated.