CF push "Failed to perform blobstore operation after three retries." for large application when using GCS Storage Account

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Operations such as cf push,  fail with the error message "Failed to perform blobstore operation after three retries."

PAS > File Storage is configured to use "External Google Cloud Storage with Service Account".

This situation is a result of a developer attempting to push large application with many dependencies which requires long time for upload.

Environment

Product Version: 2.3

Resolution

There is a known bug with the fog-google gem which affects the "Service Account" option for GCS: https://github.com/fog/fog-google/issues/323. This problem typically affects large applications with many dependencies.

The solution is to switch from using "External Google Cloud Storage with Service Account" to "External Google Cloud Storage with Access Key and Secret Key".

If you have PAS configured to use "Service Account", then it is recommended to switch to using "Access Key". There is not currently a workaround to this problem, this issue is caused by a bug in fog-google gem. A CAPI fix to this issue is pending update to fog-google gem.