Buildpacks are missing or not installed in PAS 2.3

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:
We will use the Ruby Buildpack as an example in this article. During the deployment of PAS we will see the following errors:

The Cloud Controller post-start.log shows everything executing successfully:

[2017-12-22 00:05:36+0000] + chpst -u vcap:vcap bundle exec rake buildpacks:install
[2017-12-22 00:05:39+0000] SEQUEL DEPRECATION WARNING: ...
[2017-12-22 00:05:40+0000] + [[ 0 -ne 0 ]]
[2017-12-22 00:05:40+0000] + popd
[2017-12-22 00:05:40+0000] + exit 0

The log messages in cloud_controller_ng.log show that there was an error uploading the Ruby Bulidpack.

{"timestamp":1513901227.9414902,"message":"Buildpack ruby_buildpack failed to install or update. Error: #<Faraday::ConnectionFailed wrapped=#<Net::OpenTimeout: Net::OpenTimeout>>","log_level":"error","source":"cc.background","data":{},"thread_id":47339562332440,"fiber_id":47339590905600,"process_id":12124,"file":"/var/vcap/data/packages/cloud_controller_ng/46683eb3e4b842f2d70df86eb544df41b20d252f/cloud_controller_ng/app/jobs/runtime/buildpack_installer.rb","lineno":38,"method":"rescue in perform"}
{"timestamp":1513901227.9422777,"message":"Request failed: 500: {\"error_code\"=>\"UnknownError\", \"description\"=>\"An unknown error occurred.\", \"code\"=>10001, \"test_mode_info\"=>{\"description\"=>\"Net::OpenTimeout\", \"error_code\"=>\"CF-ConnectionFailed\", \"backtrace\"=>[\"/var/vcap/packages/ruby-2.3/lib/ruby/2.3.0/net/http.rb:930:in `connect'\", \"/var/vcap/packages/ruby-2.3/lib/ruby/2.3.0/net/http.rb:863:in `do_start'\", \"/var/vcap/packages/ruby-2.3/lib/ruby/2.3.0/net/http.rb:852:in `start'\", \"/var/vcap/packages/ruby-2.3/lib/ruby/2.3.0/net/http.rb:1398:in `request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/faraday-0.11.0/lib/faraday/adapter/net_http.rb:80:in `perform_request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/faraday-0.11.0/lib/faraday/adapter/net_http.rb:38:in `block in call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/faraday-0.11.0/lib/faraday/adapter/net_http.rb:85:in `with_net_http_connection'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/faraday-0.11.0/lib/faraday/adapter/net_http.rb:33:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/faraday_middleware-0.11.0.1/lib/faraday_middleware/response/follow_redirects.rb:78:in `perform_with_redirection'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/faraday_middleware-0.11.0.1/lib/faraday_middleware/response/follow_redirects.rb:66:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/faraday-0.11.0/lib/faraday/rack_builder.rb:139:in `build_response'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/faraday-0.11.0/lib/faraday/connection.rb:377:in `run_request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/azure-core-0.1.8/lib/azure/http_response_helper.rb:27:in `set_up_response'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/azure-core-0.1.8/lib/azure/core/http/http_request.rb:143:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/azure-core-0.1.8/lib/azure/core/http/retry_policy.rb:41:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/azure-core-0.1.8/lib/azure/core/http/retry_policy.rb:41:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/azure-core-0.1.8/lib/azure/core/http/http_request.rb:104:in `block in with_filter'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/azure-core-0.1.8/lib/azure/core/http/signer_filter.rb:28:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/azure-core-0.1.8/lib/azure/core/http/signer_filter.rb:28:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/azure-core-0.1.8/lib/azure/core/http/http_request.rb:104:in `block in with_filter'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/azure-core-0.1.8/lib/azure/core/service.rb:36:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/azure-core-0.1.8/lib/azure/core/filtered_service.rb:34:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/azure-core-0.1.8/lib/azure/core/signed_service.rb:41:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/azure-storage-0.12.1.preview/lib/azure/storage/service/storage_service.rb:53:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/azure-storage-0.12.1.preview/lib/azure/storage/blob/blob_service.rb:59:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/azure-storage-0.12.1.preview/lib/azure/storage/blob/block.rb:154:in `put_blob_block'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/fog-azure-rm-0.3.1/lib/fog/azurerm/requests/storage/put_blob_block.rb:12:in `put_blob_block'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/fog-azure-rm-0.3.1/lib/fog/azurerm/requests/storage/multipart_save_block_blob.rb:74:in `block (2 levels) in multipart_save_block_blob'\"]}}","log_level":"error","source":"cc.background","data":{"job_guid":"f44f29ac-94c8-495a-8e33-55d597f14d45"},"thread_id":47339562332440,"fiber_id":47339590905600,"process_id":12124,"file":"/var/vcap/data/packages/cloud_controller_ng/46683eb3e4b842f2d70df86eb544df41b20d252f/cloud_controller_ng/app/jobs/logging_context_job.rb","lineno":43,"method":"block in log_error"}

Environment

Cause

The Cloud Controller uploads the Buildpacks in the background. Occasionally during the PAS deployment there is a Blobstore error when uploading one of the Buildpacks. This will result in a successful deployment but a missing or an older version of a Buildpack will still exist.

Resolution

1. SSH into Cloud Controller instance 0 using the BOSH CLI. You must run step 2 from cloud controller instance 0 because post start will not re-upload the buildpacks when executed on any other instance.

bosh -d cf ssh cloud_controller/0

2. Manually execute the post-start script from cloud_controller/0:

/var/vcap/jobs/cloud_controller_ng/bin/post-start; echo $?

3. The echo command should return a zero indicating post-start was successful. If the echo returns a value of 1 or higher, the script was unsuccessful.

4. After several minutes have past, observe that the missing or outdated Buildpacks are no longer present using the CF CLI.

cf buildpacks