Azure Storage Account Lockout during Tanzu Mission Control Managed Cluster Recovery following Power Event
search cancel

Azure Storage Account Lockout during Tanzu Mission Control Managed Cluster Recovery following Power Event

book

Article ID: 438024

calendar_today

Updated On:

Products

VMware Tanzu Mission Control

Issue/Introduction

Following a cluster-wide power event, Tanzu Mission Control (TMC) managed Kubernetes clusters may experience a Microsoft Azure storage account lockout.
This is characterized by TMC Data Protection backup failures and Azure API throttling.

Symptoms:

  • Data Protection logs show: rpc error: code = Unknown desc = HEAD https://<REDACTED>.blob.core.windows.net/.../velero-backup.json RESPONSE 403: 403 This request is not authorized to perform this operation. ERROR CODE: AuthorizationFailure.

  • Azure Portal indicates service principal throttling on Microsoft.Storage/storageAccounts/listKeys/action calls.

  • Backups and reconciliations fail immediately after cluster power-on.

Environment

Product: Tanzu Mission Control (TMC)
Feature: Data Protection (Velero)
Cloud Provider: Microsoft Azure
Authentication Method: Service Principal (List Keys API)

 

Cause

A "thundering herd" effect occurs during cluster initialization where simultaneous reconciliation requests from the Velero Azure plugin generate an excessive volume of "list keys" API calls, exceeding Azure's security thresholds and triggering an automated account lockout.

Resolution

A manual operational workaround is required until AAD-based authentication (Managed Identity) is fully supported within the TMC-integrated Velero version.

  1. Intercept Cluster Boot: During a managed recovery or cold boot, prevent the Velero pods from starting immediately.

  2. Scale Down Velero: Execute the following command to disable automated reconciliation: kubectl scale deployment velero -n vmware-system-tmc --replicas=0

  3. Verify Infrastructure: Ensure storage connectivity and Azure Service Principal status are healthy.

  4. Controlled Restoration: Scale the Velero deployment back to its original state only after the cluster nodes have stabilized: kubectl scale deployment velero -n vmware-system-tmc --replicas=1

  5. Monitoring: Coordinate with Microsoft Azure support to clear any existing suppression on the service principal if the 403 error persists.

Additional Information

Future releases of vSphere Kubernetes Service Manager are expected to include support for AAD (Managed Identity) authentication, which utilizes the IMDS endpoint for token acquisition and removes the requirement for "list keys" API calls.

Powered by Wolken Software