BBR failing on S3 versioned bucket as it failed to retrieve versions for objects in the bucket

Products

• Operations Manager

Issue/Introduction

BOSH Backup and Restore (BBR) job s3-versioned-blobstore-backup-restorer failed with error:

“Error attempting to run backup for job s3-versioned-blobstore-backup-restorer on backup_restore/a98bf4e8-3bc0-4f48-93f1-c917da0fcf5b: 2020/10/06 03:50:18 Failed to run: failed to retrieve versions; bucket '<BUCKET-NAME>' has `null` VerionIds - exit code 1”

BRR has identified that there are un-versioned objects in a versioned bucket and it will exit. 

BBR will only backup the "latest" or "current" version of the object.

Environment

OPS Manager with external S3 Blobstore

Resolution

Identifying why there are un-versioned objects in the versioned S3 bucket is beyond the scope of this KB.

For AWS S3 bucket you can identify what objects are "current" and "un-versioned" using the commands below.
For non-AWS S3 buckets, consult the relevant document from the Storage Provider.

Get a list of all objects in the bucket:

aws s3api list-object-versions --bucket <BUCKET-NAME>  --output json  > objects.json

Retrieve the object ID or Key of any "current" objects with VersionId=null:

cat objects.json | jq '.Versions[] | select((.IsLatest==true) and .VersionId=="null") | .Key'

Copy the object to the bucket, this will generate a VersionID for the object

aws s3 cp s3://BUCKET-NAME/OBJECT-ID  s3://BUCKET-NAME/OBJECT-ID --metadata bump=true

If there are a large number of objects, you can copy all files in the bucket. 

aws s3 cp s3://BUCKET-NAME/  s3://BUCKET-NAME/ --recursive --metadata bump=true

Once the objects have been copied, repeat the above steps, and verify all objects have a VersionID.
Lastly, run BBR.