Tanzu RabbitMQ Pods Fail to Start Due to ImagePullBackOff Error in TKGm 2.5.1

Products

• VMware Tanzu Data Suite

Issue/Introduction

In a Tanzu Kubernetes Grid (TKGm) 2.5.1 environment, the RabbitMQ cluster pods were stuck in a NotReady state. The pods were unable to pull the required container image, resulting in ImagePullBackOff errors. This blocked successful deployment of the RabbitMQ cluster.

Environment

VMware Tanzu Data Suite

VMware Tanzu RabbitMQ for Kubernetes 3.13.6

Cause

The root cause was an improperly configured imagePullSecret. Although the secret was present, it either contained an incorrect or missing authentication token. This resulted in the following error from the kubelet:

failed to authorize: failed to fetch oauth token: unexpected status from GET request ... 401

This indicates that the registry authentication failed when Kubernetes attempted to pull the image from:

rabbitmq-kubernetes.packages.broadcom.com/tanzu-rabbitmq-package-repo

Resolution

To resolve the issue, the imagePullSecret was corrected to include proper authentication details, including the "auth" field (base64 encoded string of username:password). The corrected secret looks like the following:

apiVersion: v1
kind: Secret
metadata:
  name: tanzu-rabbitmq-registry-creds   # Name of the secret
  namespace: secrets-ns                 # Replace with appropriate namespace
type: kubernetes.io/dockerconfigjson
stringData:
  .dockerconfigjson: |
    {
      "auths": {
        "rabbitmq-kubernetes.packages.broadcom.com": {
          "username": "your-username",
          "password": "your-password",
          "auth": "base64(username:password)"
        }
      }
    }

After applying this corrected secret, the RabbitMQ pods were successfully able to pull the required images and transitioned to a Running and Ready state.