Pivotal Cloud Foundry Error: "Error Performing Request: Put /v2/resource_match: net/http: Request Canceled"

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

Pushing an app fails with the following error when an external Load Balancer is in use:

Uploading console...
FAILED
Error uploading application.
Error performing request: Put /v2/resource_match: net/http: request canceled

Environment

Cause

This can be caused by an external Load Balancer having a short TCP Idle Timeout and timing out the TCP connection between CF CLI and the Cloud Controller.

When pushing an app, the CF CLI has a mechanism to avoid uploading files which already exists in the blobstore.

The follow steps are performed:

1. Generated SHA1 hash for every file in the app directory.
2. Sent the list of SHA1 hashes to the Cloud Controller URL /v2/resource_match in a single HTTP request.
   Example with 3 files:

   PUT
   /v2/resource_match
   [
     {"sha1":"013110a30e2a475551c801b4c45e497ce71c26fe","size":65582},
     {"sha1":"028a725131883dd1a9536927ad36c0616b9c95ae","size":433173},
     {"sha1":"03e63a25de4065697b185f523ed5e020c0db3ff0","size":1078479}
   ]

3. The Cloud Controller then checks if each of the files exist in the blobstore.
4. The Cloud Controller then returns a list of existing files to the CF CLI.
5. The CF CLI will only push the files which do not exist already.

If for example, the app contains 1000 files, then the Cloud Controller must make 1000 requests to the blobstore to check if each of the files exists. If the combined time for these requests is longer than the Load Balancer TCP Idle Timeout, then the HTTP request to /v2/resource_match will timeout and the net/http: request canceled error will occur.

 

Resolution

The Load Balancer should be reconfigured to increase the TCP Idle Timeout. There is no recommended value. Each environment will be different depending on the size (number of files) of the apps and the response time of the blobstore. F5 Load Balancers, for example, have a default ofTCP Idle Timeout 300 seconds (5 minutes). So increasing it to 600 seconds (10 minutes) may be a good first step. 

This parameter may have a different name and value in different load balancers. Please see the Load Balancer vendor's documentation for the specific parameters.

If reconfiguring the Load Balancer is not possible, then another option is to reduce the number of files in the app. For example, with a Java app, it may be possible to set up a local maven repository to store all the application dependencies and exclude dependencies from the JAR file.