Apply Changes Fails on Cloud Controller using Azure Storage

Products

• VMware Tanzu Application Service

Issue/Introduction

When applying changes to TAS using Azure storage, Cloud Controller might fail to start due to an error when accessing the blobstore.

In /var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log you see an AuthorizationFailure (403) error message:

{"timestamp":"<REDACTED>","message":"Error with blobstore: Fog::AzureRM::CustomAzureCoreHttpError - AuthorizationFailure (403): This request is not authorized to perform this operation.\nRequestId:<REDACTED>\nTime:<REDACTED>","log_level":"error","source":"cc.error_handling_client","data":{},"thread_id":<REDACTED>,"fiber_id":<REDACTED>,"process_id":<REDACTED>,"file":"/var/vcap/data/packages/cloud_controller_ng/<REDACTED>/cloud_controller_ng/lib/cloud_controller/blobstore/fog/error_handling_client.rb","lineno":67,"method":"rescue in error_handling"}

Cause

This error can be misleading since a 403 error code suggests a problem with your credentials, but it can also be caused by a network access issue.

Resolution

1. First check and verify your azure storage name and key are correct in Opsman -> TAS Tile -> File Storage settings.
2. If your credentials are correct, the next likely problem is the Cloud Controller is not allowed under the storage network settings. Check the Network settings under your storage account in the Azure Portal and verify the Cloud Controller IPs/subnet is allowed to access it.

Additional Information

Configuring file storage for TAS for VMs: https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/tanzu-platform-for-cloud-foundry/6-0/tpcf/pas-file-storage.html

Configure network access to Azure Storage: https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security?tabs=azure-portal#configure-network-access-to-azure-storage