System Pods in ImagePullBackOff state due to deprecated "extensions.stacks.whitesand.tmc.tanzu.broadcom.com" registry references

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• After transitioning away from Tanzu Mission Control (TMC) SaaS, users may observe the following issues within their Tanzu Kubernetes Grid (TKG) or Supervisor clusters:
  • Multiple system pods remain in an ImagePullBackOff or ErrImagePull state.
  • System components fail to initialize or recover after a restart.
  • Running kubectl get pods -A reveals persistent failures in namespaces associated with system extensions or the kapp-controller.
    
    
• Describing a pod in ImagePullBackOff states that it is failing to pull image from "extensions.stacks.whitesand.tmc.tanzu.broadcom.com"
  
  
• Describing a pkgi in ReconcileFailed sate shows the following error:
      Fetching image:
      Error while preparing a transport to talk with the registry:
        Unable to create round tripper:
          Get "https://extensions.stacks.whitesand.tmc.tanzu.broadcom.com/v2/":

Environment

vSphere Kubernetes Service

Cause

The registry URL extensions.stacks.whitesand.tmc.tanzu.broadcom.com was previously utilized by TMC SaaS for hosting system and extension images. Following the transition away from TMC SaaS, this registry is no longer accessible.

Existing deployments and PackageInstalls (pkgi) within the cluster may still contain hardcoded references to this deprecated registry. Because the container runtime cannot reach the legacy endpoint, it cannot retrieve the necessary binaries, leading to a permanent ImagePullBackOff cycle.

Resolution

Redirect the cluster to use the supported Broadcom public registry or internal repository.

• Update the Kapp-Controller - The kapp-controller manages the lifecycle of other packages. It must be updated first to ensure it can process further changes.
  • Edit the kapp-controller deployment:
    kubectl edit deployment kapp-controller -n tkg-system
  • Locate the image: field and update it to the following supported reference:
    New Image: projects.registry.vmware.com/tkg/kapp-controller:v####
    
    
• Identify and Update Failing Packages - Other extensions managed via the Package repository may also be failing.
  • Identify PackageInstall (pkgi) resources in a ReconcileFailed state: kubectl get pkgi -A | grep ReconcileFailed
  • For each failed package, identify the underlying Package (pkg) object.
  • Update the image references within the affected package definitions to point to the valid registry:
    New Registry Root: projects.registry.vmware.com/tkg/..