BBR backup/restore for TAS fails without sufficient permission for AWS S3 bucket

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

If you use AWS S3 for Cloud Controller's filesystem and try BBR backup/restore for TAS, it may fail with the following errors. The reasons have to do with insufficient permission for the target AWS S3 bucket.


Example error for BBR backup:

Error attempting to run backup for job s3-unversioned-blobstore-backup-restorer on backup_restore/89*fec*a-2*e6-*69f-82fa-*c52a*8ad*d*: 2020/04/21 11:51:34 Failed to build backups to start: could not check if bucket yourenv-buckup-bucket-1234 is versioned: AccessDenied: Access Denied
        status code: 403, request id: 6*3*0*DCC5*43*A1, host id: lkjM*cR*0qS*U*/8*K*ay4*Ekd*YXmqG0X*o*Umz*b*9u**DbVU277dU*Q*x*ls/mY*K*T*oz**o0= - exit code 1

Example error for BBR restore:

Error attempting to run restore for job s3-unversioned-blobstore-backup-restorer on backup_restore/*9**ec0a-20**-46*f-*2fa-5c5*a*8*d*da: 2020/04/21 12:40:29 Failed to run: failed to restore bucket myenv-droplets-bucket-1234: failed to get blob size for blob '2020_04_21_12_22_49/droplets/*d/e*/*d*b*d1d-27*4-*03a-b*9d-0*e*824*92*2/c*c6*7*ebee4be1279**cf7*41*b3*b8c*7*bf40' in bucket 'yourenv-buckup-bucket-1234': Forbidden: Forbidden
        status code: 403, request id: 29**05*18*5D9*5F, host id: ++/*/J*0*E3+*h*VqtYjMw1*Z*I*PjILwUHf*z*26FcPXNgO*RPvfAc*rDe2AnBK*m2uA3sUYTo=
failed to get blob size for blob '2020_04_21_12_22_49/droplets/*6/*2/3*d2*cc9-c5c5-4ec*-*ce5-498c1*0*e031/384603*b*8208e252**d*b6c*f9c7e53e66*3*8d' in bucket 'yourenv-buckup-bucket-1234': Forbidden: Forbidden
  :

Resolution

You need the following access permissions for an S3 bucket that is used for BBR backup/restore along with your IAM user for Ops Manager. This should be created manually or generated automatically by Terraform, such as pcf-user, yourenv_om_user, and so on. 
 

• For BBR backup: s3:GetBucketVersioning
• For BBR restore: s3:GetObject

The example policy is shown below:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "BBRBackupRestorePolicy",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:ListMultipartUploadParts",
                "s3:CreateBucket",
                "s3:PutObject",
                "s3:GetBucketVersioning",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::yourenv-buckup-bucket-1234/*",
                "arn:aws:s3:::yourenv-buckup-bucket-1234"
            ]
        }
    ]
}

These errors could typically happen if you use the same AWS account for Ops Manager and for the MySQL tile in order to backup data. However we recommend that you use different AWS accounts for such purposes. The minimal policy for the MySQL tile for S3 backup does not allow both s3:GetBucketVersioning and s3:GetObject. As a result, the minimal policy for the MySQL tile has an impact on BBR's access permissions if you use the same AWS account for Ops Manager and for MySQL tile.