Error: "unexpected status from GET request to https://packages.broadcom.com/artifactory/api/docker//v2/token?scope=repository%3Atkg%2Fkube-apiserver%3Apull&service=packages.broadcom.com: 404" causes Control Plane Node deployment to fail

Products

• VMware Cloud Director

Issue/Introduction

• Unable to deploy in Tanzu Kubernetes Cluster and failing during the control plane creation
• From the /root/kubeadm.err on the control plane identified the following error:
  "error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR ImagePull]: failed to pull image packages.broadcom.com/tkg/kube-apiserver:v1.30.2_vmware.1: output: E0304 15:41:47.030791    1208 remote_image.go:167] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"packages.broadcom.com/tkg/kube-apiserver:v1.30.2_vmware.1\": failed to resolve reference \"packages.broadcom.com/tkg/kube-apiserver:v1.30.2_vmware.1\": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://packages.broadcom.com/artifactory/api/docker//v2/token?scope=repository%3Atkg%2Fkube-apiserver%3Apull&service=packages.broadcom.com: 404 " image="packages.broadcom.com/tkg/kube-apiserver:v1.30.2_vmware.1"
time="2026-03-04T15:41:47Z" level=fatal msg="pulling image: rpc error: code = Unknown desc = failed to pull and unpack image \"packages.broadcom.com/tkg/kube-apiserver:v1.30.2_vmware.1\": failed to resolve reference \"packages.broadcom.com/tkg/kube-apiserver:v1.30.2_vmware.1\": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://packages.broadcom.com/artifactory/api/docker//v2/token?scope=repository%3Atkg%2Fkube-apiserver%3Apull&service=packages.broadcom.com: 404 "
, error: exit status 1"
• You have changed the CSE container registry on the CSE Server from packages.broadcom.com to projects.packages.broadcom.com

Environment

VMware Cloud Director version: 10.6.x
VMware Cloud Director Container Service Extension 4.2.3

Cause

The issue occurs when on one of the multiple CSE Servers , the CSE service was not restarted and ti will be still using the old Container registry value packages.broadcom.com.

Note: It is a requirement that changing the container registry url in the CSE Server configuration, it will require the CSE service on each CSE server to be restarted in order to apply the new configuration.

Resolution

To resolve the issue you will need to make sure that all CSE services are restarted on the CSE Servers.

Workaround

Apply the following steps:

1. On the failed Control Plane Node VM list the contents of the following YAML to see what Containeer registry is listed:

cat /run/kubeadm/kubeadm.yaml

2. If any still pointing to wrong Container Registry, Stop and start the CSE Service on ALL CSE Servers. This can be done by SSH on each CSE Server and running:

    systemctl stop cse
    
  systemctl start cse
    
3. Retry a cluster deployement and confirm what the Control Plane Node uses for the image pulls:

    tail -f /var/log/var/log/cloud-init-output.log