Backup (BBR) of the BOSH Director fails when using the IAM Instances Profile to authenticate with S3 Blobstore

Products

• Operations Manager

Issue/Introduction

An error occurs when backing up a Bosh Director using a Bosh Backup and Restore (BBR) that is configured to use S3 buckets as the Blobstore destination and IAM Instances profiles for authentication, as shown below:


The backup fails with the following error:

Error attempting to run backup for job s3-versioned-blobstore-backup-restorer on bosh/0: 2020/12/24 05:14:32 Failed to run: could not check if bucket <BUCKET_NAME> is versioned: EmptyStaticCreds: static credentials are empty
Error: failed to run job-process: exit status 1 (exit status 1) - exit code 1

Environment

Product Version: 2.10

Resolution

The issue has to do with the fact that the operator chose IAM instance profile over static credentials (aws_access_key and secret_key) for authentication. As it stands, the support for IAM instance profiles when running a BBR is hardcoded to false, forcing access to be done by static credentials only.

'use_iam_profile' => false

Permanent fix:

At the moment, there isn't a permanent fix for this issue. The VMware Tanzu team is working on a permanent fix that will allow the operator to use both static credentials and IAM Instance Profiles when authenticating with the Blobstore when running a BBR backup. 
 

Workaround:

As a workaround, the operator can hardcode the buckets.json file so that the flag use_iam_profile is set to true.
 

• SSH to the bosh director
• Switch to root
• Edit (vim/nano) the file /var/vcap/jobs/s3-versioned-blobstore-backup-restorer/config/buckets.json

{"director-blobstore": {"aws_access_key_id":null, "aws_secret_access_key":null, "endpoint":"https://s3.us-west-2.amazonaws.com", "name":"<BUCKET_NAME>", "region":"us-west-2", "use_iam_profile":false, "force_path_style":false}}

• Change the value of use_iam_profile to true.