"Failed to Perform Blobstore Operation After Three Retries" Cloud Controller Clock Configured with an Incorrect Cert Path

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

While running "cf copy-source APP1 APP2", users see the following error:

FAILED
Failed to perform blobstore operation after three retries.

From cloud controller server /var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log

{"timestamp":1488480555.86369,"message":"cp_file_between_keys-retry","log_level":"debug","source":"cc.blobstore.dav_client","data":{"request_guid":"c1674444
-cc22-4acd-572b-01b6467b1b22::1232a487-9a34-4565-a65d-ccb87da14477","error":"SSL verification failed","remaining_retries":0,"args":{"source_key":"65d57057-d
1e6-4766-a97c-89e976180770","destination_key":"b966ad4b-18a1-4072-a3ba-fe99ddd7b09f"}},"thread_id":46914699153700,"fiber_id":46914742844680,"process_id":118
45,"file":"/var/vcap/data/packages/cloud_controller_ng/5f59122e1ae729a0ec63a4d511cf61f284df9e97.1-6e01d314dfb3752e6b32c8a467467889309c34b7/cloud_controller_
ng/lib/cloud_controller/blobstore/retryable_client.rb","lineno":131,"method":"rescue in with_retries"}

This is a known issue that is causing the failure of the background jobs that interact with WebDAV. The issue also results in blobs not being deleted. The fix is being worked on now and will be backported to 1.8, 1.9. Pivotal does not currently have a procedure available for reclaiming the lost space. If this error has caused you to lose a large amount of space, please contact Pivotal Support.

 

Environment

Resolution

cloud_controller_worker config:

The file /var/vcap/jobs/cloud_controller_worker/config/cloud_controller_ng.yml referenced the following for certificates for the blobstores:

ca_cert_path: /var/vcap/jobs/cloud_controller_ng/config/certs/buildpacks_ca_cert.pem

But this is the path for cloud controller, not the cloud controller worker. The correct path for the worker should be:

ca_cert_path: /var/vcap/jobs/cloud_controller_worker/config/certs/buildpacks_ca_cert.pem

This is causing some worker tasks to fail with cert errors when accessing blobstore endpoints.

The fix is available in PCF version 1.8.33 and version 1.9.6 onwards.