cf push command is slow and takes a long time to complete when using Google Cloud Platform (GCP) for the Blobstore in Tanzu Application Service for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When the Cloud Controller Blobstore is set to Google Cloud Platform (GCP) Cloud Store, you are experiencing a sharp rise in the time it takes to complete a cf push command.

This issue is related an overburdened Cloud Controller or Cloud Controller Worker. You can use the cf firehose plugin and run the following commands to see if this is the case:

// check CPU load: if value is greater then 1, your CC is overloaded (scale horizontally)
cf nozzle -no-filter | grep "origin:\"cc\"" | grep cpu_load_avg

// check requests outstanding: If greater then 20, CC or CC workers needed to be scaled 
cf nozzle -no-filter | grep "origin:\"cc\"" | grep "requests.outstanding"

If the metrics above seem to be normal, then it is possible that the GCP blobstore is causing delays with Resource Matching.

The following screenshot shows how Healthwatch is showing high push times and when these times trigger an alert.

Environment

Product Version: 2.11

Resolution

The root cause for this issue is still undetermined. The issue stems from the CF API trying to locate artifacts that are cached and stored in the GCP blobstore.

This resource lookup or matching is taking a very long time in GCP. In Amazon Web Services (AWS), resource matching can take a long time if you have versioning enabled for the S3 buckets. We recommend turning versioning off in any case, and letting the CF API handle versioning. 

To work around this issue when using GCP Blobstore, use the following cf CLI command to turn resource matching off at the API level.

cf curl -X PATCH /v3/feature_flags/resource_matching -d  -H "Content-type: application/json"   -d '{"enabled": false}'

Note: This turns off resource matching globally. The cf push command still may run slow, as the cached artifacts are not being utilized. This is especially true for apps that don't include a large number of files in the package, but overall it should reduce the cf push times.