Tanzu RabbitMQ package reconcile fails with "Token failed verification: expired" error

Products

• VMware Tanzu Data Suite
• VMware Tanzu RabbitMQ
• RabbitMQ

Issue/Introduction

This article provides step-by-step instructions to resolve a failed reconciliation of the tanzu-rabbitmq package caused by an expired token used to access the RabbitMQ container registry.

When running the command:

tanzu package installed status -n tkg-packages tanzu-rabbitmq

You observe output similar to this:

Reconcile failed

Fetching image:
Error while preparing a transport to talk with the registry:
GET https://rabbitmq-kubernetes.packages.broadcom.com/... : Token failed verification: expired

Cause

The tanzu-rabbitmq package fetches container images from a private Broadcom registry which uses a token-based authentication mechanism. These tokens are valid for 6 months. When the token expires, image pull requests fail, causing the package reconciliation to continuously fail.

Resolution

 

Step1: Get a New Token from Broadcom

• Go to https://support.broadcom.com/

• Navigate to My Downloads (left sidebar).

• Search for RabbitMQ.

• Select VMware Tanzu RabbitMQ for Kubernetes.

• Click the Token Download icon to copy the new access token.

  • Note: The token is valid for 6 months.

 

Step2: Update the image pull secret

• Backup the existing secret:

  kubectl get secret tanzu-rabbitmq-registry-creds -n tkg-packages -o yaml > backup-rabbitmq-secret.yaml

• Delete the old secret:

   

  kubectl delete secret tanzu-rabbitmq-registry-creds -n tkg-packages

• Create a new secret using the updated token:

   

  kubectl create secret docker-registry tanzu-rabbitmq-registry-creds -n tkg-packages \
  --docker-server=rabbitmq-kubernetes.packages.broadcom.com \
  --docker-username=<your-email> \
  --docker-password=<new-token> \

  Replace <your-email> and <new-token> with the appropriate values.

 

Step3: Let the reconciliation retry automatically

• Verify with below command if the reconcile failed error is gone:

  tanzu package installed list -n tkg-packages