cloud_controller_ng fails to start when it is not able to connect to blobstore

Products

• VMware Tanzu Application Service

Issue/Introduction

• The issue usually occurs with external blobstore. 
• `monit summary` shows cloud_controller_ng is running, but nginx_cc is failing.
• In cloud_controller_ng.log, the line before prior to the error is database schema check, then it takes around 5 minutes to see error "Error with blobstore: Excon::Error::Timeout". 

...
{"timestamp":"2025-02-04T14:12:24.4143479632Z","message":"database schema is as new or newer than locally available migrations","log_level":"info","source":"cc.db.wait_until_current","data":{},"thread_id":45134,"fiber_id":36540,"process_id":8,"file":"/var/vcap/data/packages/cloud_controller_ng/e6b2097ed0a18d4c5c39195ffbc9eff588f04cff/cloud_controller_ng/lib/cloud_controller/db_migrator.rb","lineno":68,"method":"wait_for_migrations!"}

{"timestamp":"2025-02-04T14:17:52.889151344Z","message":"Error with blobstore: Excon::Error::Timeout - connect_write timeout reached","log_level":"error","source":"cc.error_handling_client","data":{},"thread_id":36520,"fiber_id":36540,"process_id":7,"file":"/var/vcap/data/packages/cloud_controller_ng/e6b2097ed0a18d4c5c39195ffbc9eff588f04cff/cloud_controller_ng/lib/cloud_controller/blobstore/fog/error_handling_client.rb","lineno":67,"method":"rescue in error_handling"}
...

 

 

Environment

Tanzu Platform for Cloud Foundry - all releases

Cause

The problem is caused by TCP connection failure to external blobstore, during cloud_controller_ng initialization, it reaches out to blobstore, but no response comes back. The Excon library will wait around 5 minutes until timeout. In this case run `perform_blobstore_benchmarks` under /var/vcap/jobs/cloud_controller_ng/bin will get similar errors. 

Resolution

As first step, please try `nc -zv <BLOBSTORE_HOSTNAME> <PORT>` to test the network connection If it fails or has no response, please troubleshoot the networking layer to resolve any proxy, firewall or other components which could block the communication.