CloudController::Blobstore::BlobstoreError: invalid: Cannot insert legacy ACL for an object when uniform bucket-level access is enabled

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Prerequisites

• File Storage for PAS is configured to use Google's Cloud Storage bucket(s)
• In Google Cloud Storage browser, you can see that bucket(s) used by TAS are set to use Uniform Access-control

Attempting to run apply changes on TAS tile fails with the following error:

Updating instance cloud_controller: cloud_controller/<instance-guid> (0) (canary) (00:06:48) L Error: Action Failed get_task: Task <agent-id> result: 1 of 2 post-start scripts failed. Failed Jobs: cloud_controller_ng. Successful Jobs: bosh-dns.

Further looking into the logs of post-start scripts (on the cloud_controller/<instance-guid> VM from the error above and under /var/vcap/sys/log/cloud_controller_ng/post-start.stderr.log), we see the following errors:

+ main
+ install_buildpacks
+ pushd /var/vcap/packages/cloud_controller_ng/cloud_controller_ng
+ chpst -u vcap:vcap bundle exec rake buildpacks:install
rake aborted!
CloudController::Blobstore::BlobstoreError: invalid: Cannot insert legacy ACL for an object when uniform bucket-level access is enabled. Read more at https://cloud.google.com/storage/docs/uniform-bucket-level-access.

Environment

Product Version: 2.7

Resolution

This issue happens when any Google Cloud Storage bucket uses or changed to using Uniform Access-control (Object access is controlled entirely through bucket-level permissions (IAM), ensuring uniform access to all objects in a bucket) when Fine-grained Access-control is required.

There are 2 options to resolve this problem:

• Option 1
  • Change the bucket access-control to fine-grained using GCP console and run the apply changes again.
• Option 2
  • Note: Follow this option if the current bucket level is Uniform and the time limit of 90 days has already passed (If a bucket's access-control is Uniform and 90 days have passed then Google prevents you to change the access-control level to fine-grained)
    • Steps:
      • Create a new bucket(s)
      • Copy over the contents from the old bucket (problematic and has Uniform Access Control) to the new bucket.
      • Respective of the new bucket, make necessary changes to tile configurations in the Ops Manager UI - For the example mentioned in the Issue section above, buildpacks bucket was the one which had Uniform access-control. After copying the contents to the new bucket, bucket name was updated under Ops Manager UI -> PAS/TAS tile -> File Storage -> External Google Cloud Storage with service account -> Buildpacks bucket name 
      • Run the Apply Changes again.