How to install and use BOSH Backup and Restore S3 bucket configuration validator in Tanzu Application Service for VMs

Products

VMware Tanzu Application Service for VMs

Issue/Introduction

The BOSH Backup and Restore (BBR) S3 bucket configuration validator is a tool to validate and troubleshoot your Cloud Foundry and BBR external blobstore configuration.

This tool looks at your bucket configuration file and validates that all credentials, buckets, and policies are in order. If the tool does not detect any issues, your backups and restores with BBR should run successfully. If the tool does detect any issues, it helps you debug the issue.

The tool is geared towards bucket configuration files that are consumed by the BBR SDK. For more information, refer to Bucket configuration files.

Both versioned and unversioned S3-compatible blobstores are supported. For more information, refer to Using versioning in S3 buckets.

Environment

Product Version: 2.10

Resolution

Installation

Note: The tool should be used on the VM that you run your backups and restores from. This is because that's where your bucket configuration file will be put by the jobs configured through the BBR SDK, and to provide realistic network conditions.

You can use the BOSH CLI to get the tool onto that VM assuming that it's pointing to your environment:

1. Run this command to find the deployment:

bosh deployments

2. Run this command to copy the binary onto the backup and restore VM:

bosh --deployment <deployment> scp bbr-s3-config-validator-linux-amd64 backup_restore:/tmp

3. SSH onto the backup and restore VM:

bosh --deployment <deployment> ssh backup_restore

4. Move the binary into your home directory to execute it:

mv /tmp/bbr-s3-config-validator-linux-amd64

Usage

By default, the tool validates a versioned buckets configuration. It expects to find this configuration at: /var/vcap/jobs/s3-versioned-blobstore-backup-restorer/config/buckets.json.

You can also use the tool to validate unversioned buckets by using the --unversioned flag.

1. To learn more about the tool usage:

./bbr-s3-config-validator-linux-amd64 --help

2. Use this command to run the tool:

./bbr-s3-config-validator-linux-amd64

The tool will run a series of tests:

Verify it can reach the blobstore and bucket.
Verify that the bucket is versioned or unversioned.
Verify it can get objects and objects metadata.
If you use the --validate-put-object flag, verify it can write an object to the bucket.

You can override the default configuration location with the BBR_S3_BUCKETS_CONFIG environment variable. This allows you to validate a configuration that you wish to apply without overriding the current configuration.

Bucket configuration files

A BBR bucket configuration file is expected to look similar to this:

{
    "some-resource-to-backup": {
        "aws_access_key_id": "<the buckets' s3-compatible blobstore's access key>",
        "aws_secret_access_key": "<the buckets' s3-compatible blobstore's secret key>",
        "endpoint": "<the s3-compatible blobstore's endpoint>",
        "name": "<the live bucket's name>",
        "region": "<the live bucket's region>",
        "backup": {
            "name": "<the backup bucket's name>",
            "region": "<the backup bucket's region>"
        }
    },
    "another-resource-to-backup": {
        ...
    },
    ...
}

Sample output

Make sure to run this command on your backup and restore VM.

$ ./bbr-s3-config-validator --validate-put-object

Validate unversioned S3 buckets configuration at:

/var/vcap/jobs/s3-unversioned-blobstore-backup-restorer/config/buckets.json

  {
    "packages": {
      "name": "packages-live",
      "region": "eu-west-1",
      "aws_access_key_id": "<redacted>",
      "aws_secret_access_key": "<redacted>",
      "endpoint": "https://s3.eu-west-1.amazonaws.com",
      "backup": {
        "name": "packages-backup",
        "region": "eu-west-1"
      }
    },
    "buildpacks": {
      "name": "buildpacks-live",
      "region": "eu-west-1",
      "aws_access_key_id": "<redacted>",
      "aws_secret_access_key": "<redacted>",
      "endpoint": "https://s3.eu-west-1.amazonaws.com",
      "backup": {
        "name": "buildpacks-backup",
        "region": "eu-west-1"
      }
    }
  }

Validating packages' live bucket packages-live ...
* Bucket is not versioned ... Yes
* Can list objects ... Yes
* Can get objects ... Yes
* Can put objects ... Yes

Validating packages' backup bucket packages-backup ...
* Bucket is not versioned ... Yes
* Can list objects ... Yes
* Can get objects ... Yes
* Can put objects ... Yes

Validating buildpacks' live bucket buildpacks-live ...
* Bucket is not versioned ... Yes
* Can list objects ... Yes
* Can get objects ... Yes
* Can put objects ... Yes

Validating buildpacks' backup bucket buildpacks-backup ...
* Bucket is not versioned ... No [reason: bucket buildpacks-backup is versioned]
* Can list objects ... Yes
* Can get objects ... Yes
* Can put objects ... Yes

Bad config
exit 1

For more information about BBR and S3 bucket configuration validator , refer to the following resources:

Additional Information

https://github.com/cloudfoundry-incubator/bosh-backup-and-restore
https://github.com/cloudfoundry-incubator/bosh-backup-and-restore/tree/master/s3-config-validator