Antrea Interworking pods fail with ImagePullBackOff error after TKG Guest cluster upgrade

Products

• VMware NSX
• VMware Cloud Foundation

Issue/Introduction

After upgrading a Tanzu Kubernetes Grid (TKG) guest cluster (e.g., from version 1.32.7 to 1.32.10) on Photon VKS 3.5, the Antrea interworking pods fail to start. The pods remain in ImagePullBackOff or ErrImagePull status. This occurs because the cluster configuration incorrectly points to a local registry (localhost:5000) rather than the official Broadcom repository.

Reviewing the pod events or logs reveals the following error:

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Warning  Failed     35m (x4247 over 15d)   kubelet            Failed to pull image "localhost:5000/tkg/packages/core/antrea@sha256:a#######113cf####16337###7#6c": rpc error: code = NotFound desc = failed to pull and unpack image "localhost:5000/tkg/packages/core/antrea...": not found

Environment

• VMware Cloud Foundation (VCF)
• VMware NSX 
• Tanzu Kubernetes Grid (TKG) 
• Photon VKS 

Cause

A race condition during the TKG guest cluster upgrade process causes the Antrea PackageInstall (PKGI) to fail its reconciliation. The system maintains a stale configuration state where the deployment attempts to pull images from localhost:5000.

Resolution Manually recreate the interworking deployment to trigger a clean reconciliation from the primary Broadcom repository.

Resolution

To resolve this issue, manually recreate the interworking deployment to trigger a clean reconciliation from the primary Broadcom repository.

1.  Connect to the affected TKG guest cluster context.

2.  Back up the existing interworking deployment

kubectl get deployment -n vmware-system-antrea interworking -o yaml > interworking-backup.yaml

3.  Delete the stale deployment

kubectl delete deployment -n vmware-system-antrea interworking

4.  Force a system reconciliation by patching the Antrea PackageInstall syncPeriod

kubectl patch pkgi -n vmware-system-tkg cluster-antrea --type='merge' -p '{"spec":{"syncPeriod":"9m"}}'

5.  Verify that the pods return to a **Running** state

kubectl get pods -n vmware-system-antrea

 

Additional Information

To speak with a customer representative or a Support Engineer see Contact Support. Scroll to the bottom of the page and click on your respective region.