Migrating AAKE Docker Image Pulls from gcr.io to packages.broadcom.com

Products

• CA Automic Workload Automation - Automation Engine
• Automic Automation

Issue/Introduction

Broadcom is retiring the gcr.io Docker registry. Kubernetes deployments (AAKE) with values.yaml overrides pointing to gcr.io/esd-automic-saas/... must be migrated to packages.broadcom.com before the decommissioning date.

Failure to migrate will result in the following symptoms for new or restarted pods:

• ImagePullBackOff errors
• ErrImagePull errors

Please read the full KB ahead of migrating. Especially if you don't want to take a backup of your database during the migration as operator.pauseOnUpgrade will pause the process. Being unprepared for it or it may introduce unnecessary downtime.

Read more about the pauseOnUpgrade here: https://techdocs.broadcom.com/us/en/ca-enterprise-software/intelligent-automation/automic-automation/26-0/automic-automation/Content-Installation_Containers-containers_Upgrading.html

Important Note: Replacing your HELM charts might not be sufficient if you have custom configurations. Make sure you also adapt any custom settings in values.yaml that overrule the default settings of the values.yaml file packaged in the HELM chart. Please review or update your custom configuration to point repository location values to the packages.broadcom.com registry.

Environment

• Product: AAKE (Automic Automation in Kubernetes)
• Affected versions: Helm chart automic-automation-6.0.0.tgz and all previous versions.

Their corresponding HELM charts for these releases already point to these docker images for fresh downloads.

• HELM 6.0.0
• HELM 4.4.5

Manual re-configuration of HELM charts that have already been downloaded for these versions (HELM 4.4.4, 4.4.4 HF1, 4.4.4 HF2, 4.4.5, 6.0.0) is possible as well.

Cause

Broadcom is consolidating Automic-branded portals into the Broadcom Support Portal. The migration involves moving from Google Container Registry (gcr.io) to a Broadcom-hosted Artifactory instance (packages.broadcom.com).

Authentication via gcr.io used GCP service account JSON keys. packages.broadcom.com uses JWT tokens via JFrog Artifactory. You must update both the registry URL in values.yaml and the Kubernetes image pull secret.

gcr.io remains available for 90 days after the legacy portal shutdown to allow for transition.

Old docker image path in HELM chart

• repository location:
  gcr.io/esd-automic-saas/automic/component/release/ae/
• example tags:
  gcr.io/esd-automic-saas/automic/component/release/ae/cp:26.0.0gcr.io/esd-automic-saas/automic/component/release/automation_ai/automation-ai:26.0.0

New docker image path in HELM chart

• repository location:
  automic-docker.packages.broadcom.com/automic/component/release/ae/
• Example tags:
  automic-docker.packages.broadcom.com/automic/component/release/ae/cp:26.0.0
  automic-docker.packages.broadcom.com/automic/component/release/automation_ai/automation-ai:26.0.0

 

This affects on-prem/customer-managed Kubernetes deployments (AAKE) pulling directly from gcr.io; SaaS-hosted instances are unaffected.

Resolution

Choose a Migration Path

Per official guidelines, there are two supported options:

 

 

Option	Description	Use Case
Mirror to private registry	Pull images to a registry you control; update values.yaml to point there.	Recommended for Production on-prem to decouple from Broadcom's registry lifecycle. Note: If your values.yaml and Kubernetes deployments already point to your own private registry, no immediate change is required. However, you must pull newer versions from automic-docker.packages.broadcom.com and push them to your private registry for future upgrades.
Reconfigure to Broadcom Registry	Update values.yaml and pull secrets to use automic-docker.packages.broadcom.com directly.	Faster execution; ideal for lab/test environments.

Phase 1: Preparation

1. Generate a Registry Token: Navigate to the Broadcom Support Portal - Registry Tokens and generate a token. Your token remains valid for the duration of your active support contract.

• • Follow KB : Registry Software Download functionality enhancements within the Broadcom Support Portal

2. Verify Connectivity: Confirm your network can reach the new registry path using your token saved in a file:

# Test using Docker
docker login automic-docker.packages.broadcom.com --username <email> --password-stdin < <token-file>
docker manifest inspect automic-docker.packages.broadcom.com/automic/component/release/ae/install-operator:<version>

# Alternatively, test using curl

curl -u "<email>@broadcom.com:$(cat token-file)" https://automic-docker.packages.broadcom.com/artifactory/api/system/ping

Phase 2: Update Configuration

Update the repository.location in your values.yaml so that it overrides to use the new registry host. The path suffix remains identical.

 

 

Field	New Value
repository.location	automic-docker.packages.broadcom.com/automic/component/release/ae/
images.automation-ai.repository	automic-docker.packages.broadcom.com/automic/component/release/automation_ai/

Phase 3: Update Kubernetes Image Pull Secret

Choose the method appropriate for your environment.

Option A: Create a brand-new secret

kubectl --namespace <your namespace> create secret docker-registry automic-image-pull-secret \
  --docker-server=automic-docker.packages.broadcom.com \
  --docker-username=<your Broadcom user id> \
  --docker-password="<your Registry token>" \
  --docker-email=<optional>

Option B: Update an existing secret This merges the new credentials into your existing secret without overwriting other registries. This reads the current secret's decoded contents first, then adds a new entry to its auths map:

ARTIFACTORY_SERVER="automic-docker.packages.broadcom.com"
ARTIFACTORY_USERNAME=<your Broadcom user id / email address>
ARTIFACTORY_TOKEN=<your Registry token>
NAMESPACE="<your Kubernetes namespace>"


kubectl --namespace $NAMESPACE get secret automic-image-pull-secret -o jsonpath='{.data.\.dockerconfigjson}' | base64 --decode > config.json

AUTH=$(echo -n "$ARTIFACTORY_USERNAME:$ARTIFACTORY_TOKEN" | base64 -w 0)

jq --arg server "$ARTIFACTORY_SERVER" \
   --arg auth "$AUTH" \
  '.auths[$server] = {"auth": $auth}' \
  config.json > updated_config.json

kubectl --namespace $NAMESPACE create secret generic automic-image-pull-secret \
  --from-file=.dockerconfigjson=updated_config.json \
  --type=kubernetes.io/dockerconfigjson \
  --dry-run=client -o yaml | kubectl --namespace $NAMESPACE apply -f -

The old gcr.io entry stays until manually removed. If this secret is shared and something else in the namespace still needs to pull from gcr.io, that keeps working. If not, the old GCP credential just sits there unused, decoded in plaintext on disk temporarily (config.json) until you delete those intermediate files.

Option C: Overwrite secret This is less ideal if you have multiple deployments of AAKE or if you share the secret with e.g. a containerized unix agent, it builds a brand-new .dockerconfig json from scratch containing only the one server you specify, then kubectl apply overwrites the whole secret object with it:

kubectl create secret docker-registry <real-pull-secret-name> \  
--docker-server=automic-docker.packages.broadcom.com \  
--docker-username=<email> \  
--docker-password="$(cat <token-file>)" \  
--namespace=<namespace> \  
--dry-run=client -o yaml | kubectl apply -f -

Phase 4: Execute Upgrade

Run the standard helm upgrade command with your updated charts and values files.

Run the upgrade:

helm upgrade <release-name> <chart-package> -f <values-file> --namespace <namespace>

Mandatory Checkpoint: If operator.pauseOnUpgrade: true is set, the operator will pause and request a database backup. helm automic-automation update --continue should be enough (doc) otherwise see additional information for a workaround.

Phase 5: Verification

Verify that pods are pulling from the new host:

kubectl get pods -n <namespace> -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}'

 

 

Additional Information

Ingress 404 Errors after Upgrade

If your pods are healthy (Ready: True) but the URL returns a 404 Not Found / nginx page, the ingress objects are likely missing the ingressClassName.

1. Check current status:

   kubectl get ingress awi jcp-rest jcp-ws -n <namespace> -o jsonpath='{.spec.ingressClassName}'

2. Verify your class name:

   kubectl get ingressclass

3. Patch the ingress objects:

   kubectl patch ingress awi -n <namespace> --type=merge -p '{"spec":{"ingressClassName":"<class-name>"}}'
   kubectl patch ingress jcp-rest -n <namespace> --type=merge -p '{"spec":{"ingressClassName":"<class-name>"}}'
   kubectl patch ingress jcp-ws -n <namespace> --type=merge -p '{"spec":{"ingressClassName":"<class-name>"}}'

Note: For a durable fix, uncomment ingressClassName in your values.yaml and run helm upgrade again.

Migration Paths

• Mirroring (Recommended for Production): Pull images to a private registry you control. This decouples your environment from Broadcom’s registry lifecycle.
• Direct Connect: Configure your cluster to pull directly from automic-docker.packages.broadcom.com. Best for lab/test environments.

Important Notes

ZDU (Zero Downtime Upgrade): Not supported for registry swaps. Use standard Helm upgrade (restart-mode).

ZDU requires upgrading to a higher Automic version. If you do not want to upgrade to a higher Automic version we recommend running a “standard” upgrade with downtime.

ZDU Upgrade Examples:
24.4.4 -> 24.4.4 (Same version / registry update only): Not supported via ZDU
24.4.4 -> 24.4.4 HFx = (Hotfix update): Supported via ZDU
24.4.4 HF1 -> 24.4.4 HF3 = (Hotfix increment): Supported via ZDU
24.4.4 -> 24.4.5 =(Minor version upgrade): Supported via ZDU 

Notes about pauseOnUpgrade:

If you have set it to false, and it is still pausing or you ran  helm automic-automation update --continue and the upgrade is still paused:

1. Back up Automation Engine, Analytics, and Automation AI databases.
2. Identify the breakpoint 

   kubectl get cm <configmap-name> -n <namespace> -o jsonpath='{.metadata.labels.features\.aa-install-operator/command}'

3. Resume by patching the ConfigMap (replace <breakpoint-name> with the text after pauseAt.):

kubectl patch cm <configmap-name> -n <namespace> --type merge --patch '{"metadata":{"labels":{"features.aa-install-operator/command":"resumeFrom.<breakpoint-name>"}}}'