BOSH Azure CPI returns HTTP 403 AuthorizationFailure when deleting unused stemcells

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

Even though BOSH Director blobstore is configured to use azure  STORAGE_ACCOUNT_ALPHA in the Director Tile bosh clean up fails when using STORAGE_ACCOUNT_BETA.   This maybe unexpected given BOSH is not configured to use STORAGE_ACCOUNT_BETA.

Example Error Message

Deleting unused stemcell 'bosh-stemcell-GUID'... Failed (00:00:02)
Cleaning up rendered CPI jobs... Finished (00:00:00)
Deleting stemcell from cloud:
  CPI 'delete_stemcell' method responded with error: CmdError

{
  "type":"Bosh::Clouds::CloudError",
  "message":"get_blob_properties: #\u003cAzure::Core::Http::HTTPError:2500 @http_response: #\u003cAzure::Core::Http::HttpResponse:0x0000... @on_complete_callbacks=[], @env=#\u003cFaraday::Env @method=:head @url=#\u003cURI::HTTPS https://STORAGE_ACCOUNT_BETA.blob.core.windows.net/stemcell/bosh-stemcell-GUID.vhd ... \"x-ms-error-code\"=\u003e\"AuthorizationFailure\", \"date\"=\u003e\"Fri, 14 Nov 2025 14:46:04 GMT\"} @status=403 @reason_phrase=\"This request is not authorized to perform this operation.\" ... ",
  "ok_to_retry":false
}
Exit code 1

 

 

Environment

Tanzu Operations Manager Version 3.1.2

Cause

When deleting unused stemcells BOSH Azure CPI will first issue rest api call Storage Accounts - List to obtain a list of storage accounts.  Once it has a list of storage accounts it will iterate over them to check if the account has a stemcell it can delete. 

Resolution

 

In some cases there was a recent change to firewall or network policies associated with one or more storage accounts access.  BOSH Director will need access to look for stemcell blobs in storage accounts returned by the Storage account list for the given resource group.