BOSH Director on Azure fails to delete unused stemcell
Article ID: 397393
Updated On:
Products
VMware Tanzu Application Service
Issue/Introduction
BOSH Director on Azure fails to delete unused stemcell with 403 not authorized error.
The BOSH deploy reports error -
Deleting unused stemcell 'bosh-stemcell-123456-########################'... Failed (00:00:02)
Cleaning up rendered CPI jobs... Finished (00:00:00)
Deleting stemcell from cloud: CPI 'delete_stemcell' method responded with error:
CmdError{"type":"Bosh::Clouds::CloudError","message":"get_blob_properties: #\u003cAzure::Core::Http::HTTPError:2500 @http_response: #\u003cAzure::Core::Http::HttpResponse:0x00007f0396a14258 @http_response=#\u003cFaraday::Response:0x00007f03969fbeb0 @on_complete_callbacks=[], @env=#\u003cFaraday::Env @method=:head @url=#\u003cURI::HTTPS https://claimsblobstore.blob.core.windows.net/stemcell/bosh-stemcell-68383d93-830c-4102-9196-aeb55c3b6409.vhd\u003e @request=#\u003cFaraday::RequestOptions open_timeout=60\u003e @request_headers={\"User-Agent\"=\u003e\"Azure-Storage/2.0.3-2.0.4 (Ruby 3.3.7-p123; Linux linux)\", \"x-ms-date\"=\u003e\"Thu, 08 May 2025 06:22:46 GMT\", \"x-ms-version\"=\u003e\"2018-11-09\", \"DataServiceVersion\"=\u003e\"1.0;NetFx\", \"MaxDataServiceVersion\"=\u003e\"3.0;NetFx\", \"Content-Type\"=\u003e\"application/atom+xml; charset=utf-8\", \"x-ms-client-request-id\"=\u003e\"f7273272-f6b8-4086-912b-6d034679f98c\", \"Content-Length\"=\u003e\"0\", \"Authorization\"=\u003e\"SharedKey claimsblobstore:pl+rnZ+PNUljIcpHW2cSfLgU28IHUEzWgSFUE/uYE3o=\"} @ssl=#\u003cFaraday::SSLOptions verify=true\u003e @response=#\u003cFaraday::Response:0x00007f03969fbeb0 ...\u003e @response_headers={\"transfer-encoding\"=\u003e\"chunked\", \"server\"=\u003e\"Microsoft-HTTPAPI/2.0\", \"x-ms-request-id\"=\u003e\"ac01f80b-e01e-0032-47e1-bfb325000000\", \"x-ms-error-code\"=\u003e\"AuthorizationFailure\", \"date\"=\u003e\"Thu, 08 May 2025 06:22:45 GMT\"} @status=403 @reason_phrase=\"This request is not authorized to perform this operation.\" @response_body=\"\"\u003e\u003e, @uri=#\u003cURI::HTTPS https://claimsblobstore.blob.core.windows.net/stemcell/bosh-stemcell-68383d93-830c-4102-9196-aeb55c3b6409.vhd\u003e\u003e, @uri: #\u003cURI::HTTPS https://claimsblobstore.blob.core.windows.net/stemcell/bosh-stemcell-68383d93-830c-4102-9196-aeb55c3b6409.vhd\u003e, @status_code: 403, @type: \"Unknown\", @description: \"This request is not authorized to perform this operation.\"\u003e\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/core/http/retry_policy.rb:58:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/core/http/http_request.rb:111:in `block in with_filter'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/core/http/signer_filter.rb:28:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/core/http/signer_filter.rb:28:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/core/http/http_request.rb:111:in `block in with_filter'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/core/service.rb:36:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/core/filtered_service.rb:34:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/core/signed_service.rb:41:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/storage/common/service/storage_service.rb:60:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-blob-2.0.3/lib/azure/storage/blob/blob_service.rb:179:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-blob-2.0.3/lib/azure/storage/blob/blob.rb:169:in `get_blob_properties'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/storage/blob_manager.rb:190:in `block in get_blob_properties'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/storage/blob_manager.rb:330:in `block in _initialize_blob_client'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/storage/blob_manager.rb:318:in `synchronize'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/storage/blob_manager.rb:318:in `_initialize_blob_client'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/storage/blob_manager.rb:187:in `get_blob_properties'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/stemcell/stemcell_manager2.rb:47:in `has_stemcell?'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/stemcell/stemcell_manager2.rb:37:in `block in delete_stemcell'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/stemcell/stemcell_manager2.rb:34:in `each'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/stemcell/stemcell_manager2.rb:34:in `delete_stemcell'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/cloud.rb:112:in `block (2 levels) in delete_stemcell'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/telemetry/telemetry_manager.rb:71:in `monitor'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/cloud.rb:108:in `block in delete_stemcell'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/bosh_common-2.0.0/lib/common/thread_formatter.rb:50:in `with_thread_name'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/cloud.rb:107:in `delete_stemcell'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/bosh_cpi-2.6.0/lib/bosh/cpi/cli.rb:90:in `public_send'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/bosh_cpi-2.6.0/lib/bosh/cpi/cli.rb:90:in `run'\n/var/vcap/packages/bosh_azure_cpi/bin/azure_cpi:36:in `\u003ctop (required)\u003e'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.6/lib/bundler/cli/exec.rb:59:in `load'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.6/lib/bundler/cli/exec.rb:59:in `kernel_load'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.6/lib/bundler/cli/exec.rb:23:in `run'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.6/lib/bundler/cli.rb:452:in `exec'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.6/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.6/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.6/lib/bundler/vendor/thor/lib/thor.rb:538:in `dispatch'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.6/lib/bundler/cli.rb:35:in `dispatch'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.6/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.6/lib/bundler/cli.rb:29:in `start'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.6/exe/bundle:28:in `block in \u003ctop (required)\u003e'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.6/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.6/exe/bundle:20:in `\u003ctop (required)\u003e'\n/var/vcap/packages/azure-cpi-ruby-3.3/bin/bundle:25:in `load'\n/var/vcap/packages/azure-cpi-ruby-3.3/bin/bundle:25:in `\u003cmain\u003e'","ok_to_retry":false} Exit code 1 ===== 2025-05-08 06:22:47 UTC Finished "/usr/local/bin/bosh --no-color --non-interactive --tty create-env /var/tempest/workspaces/default/deployments/bosh.yml -l /var/tempest/workspaces/default/deployments/vars.yml --package-dir /var/vcap/packages"; Duration: 18332s; Exit Status: 1
Resolution
This problem may be due to a change in storage accounts on Azure. It may require updating bosh-state.json file on OpsManager.
1.) SSH into OpsManager VM
2.) Create a backup of bosh-state.json
cp /var/tempest/workspaces/default/deployments/bosh-state.json ~/bosh-state.json.bak
3.) Find the stemcell in bosh-state.json mentioned in deploy failure message - Deleting unused stemcell 'bosh-stemcell-123456-########################'
vim /var/tempest/workspaces/default/deployments/bosh-state.json
"stemcells": [
{
"id": "#########-####-####-####-############",
"name": "bosh-vsphere-esxi-ubuntu-jammy-go_agent",
"version": "1.508",
"api_version": 3,
"cid": "sc-123456-##########################"
},
{
"id": "######-####-####-####-############",
"name": "bosh-vsphere-esxi-ubuntu-jammy-go_agent",
"version": "1.708",
"api_version": 3,
"cid": "sc-##################################"
}
],
4.) Remove the unused stemcell block from bosh-state.json. In the above example, you would delete these lines -
{
"id": "#########-####-####-####-############",
"name": "bosh-vsphere-esxi-ubuntu-jammy-go_agent",
"version": "1.508",
"api_version": 3,
"cid": "sc-123456-##########################"
},
WARNING: Make sure to only remove the stemcell mentioned by Ops Manager as "unused" and not any active stemcells. It's recommended to validate bosh-state.json is a valid json file after edit.
5.) Retry the apply changes on OpsManager and it should no longer encounter error. You may also need to manually clean up stemcell VM with removed ID on Azure side.
Feedback
Yes
No
Powered by
