CCI Supervisor Service deployment fails due to image pull errors when deployed from a private registry

Products

VMware vSphere Kubernetes Service

Issue/Introduction

CCI Supervisor Service fails to deploy because of pods not starting

Errors will be displayed based on the failure in connectivity in the pod describe :

Message:          failed to pull images: failed to get images: Image svc-cci-service-domain-<cluster id>/cci-namespace-ui-se-11e493####7d2dce3b7ce07-v62351 has failed. Error: Failed to resolve on node esx##.domain.com. Reason: Http request failed. Code 400: ErrorType(2) failed to do request: Head "https://projects.packages.broadcom.com/v2/vcf_cci_service/cci-namespace-ui-service/manifests/sha256:14f7a357002a5d4935b65f4418b52366bfc76aa96b2cd23226c97af725d82fce": <communication error> : ErrImagePull

OR

failed to get images: Timed out with error: Image svc-cci-service-domain-<cluster id>/cci-namespace-ui-se-11e493275cf####dce3b7ce07-v41020 not ready. State: Created Resolver node:

The private registry in use could be JFrog's Artifactory or any other private registry

Environment

vSphere with Tanzu 8.x with private registry

Cause

The CCI Service bundle contains image references that points to external repositories instead of references to images hosted within the same repository.
As a result, these images are not automatically rewritten or pulled correctly by Artifactory during mirroring or deployment.

Resolution

The LCI service serves as an alternative to CCI and includes the fix.

If using CCI is not an option and CCI is explicitly required, you can follow the below workaround for the issue:

From a workstation with internet access, authenticate to both the public Broadcom registry and the internal registry. Ensure the imgpkg CLI tool is installed.
Locate the exact bundle path within the downloaded CCI YAML file (typically found under the imgpkgBundle definition).
Copy the bundle to the internal registry: imgpkg copy -b projects.packages.broadcom.com/v2/<CCI_BUNDLE_PATH> --to-repo <INTERNAL_REGISTRY_FQDN>/<PROJECT_PATH>
Open the downloaded CCI Supervisor Service YAML file in a text editor.
Search for all instances of projects.packages.broadcom.com and replace them with the newly copied internal registry path: <INTERNAL_REGISTRY_FQDN>/<PROJECT_PATH>.
Save the modified YAML file.
In the vSphere Client, navigate to Workload Management > Services.
Remove the currently failed CCI service deployment and delete the corresponding Service Version.
Add a new Service Version by uploading the modified YAML file.
Install the CCI service on the Supervisor cluster using the newly uploaded version.

Additional Information

Make sure you have copied images using the below documentation if you are using Harbor
Deploying Supervisor Services from a Private Container Image Registry

The LCI service has all the references in the same repository / bundle

# /usr/lib/vmware-wcp/tools/imgpkg pull -b projects.packages.broadcom.com/vsphere/iaas/lci-service/9.0.0-beta/lci-service:9.0.0-8594cb6b  -o ./lci-package ; grep -r image: lci-package/

Pulling bundle 'projects.packages.broadcom.com/vsphere/iaas/lci-service/9.0.0-beta/lci-service@sha256:f2e4438c365714e39b4bb955dfba5b344ed020f7f63a3c89474169a6b6b317fa'
  Extracting layer 'sha256:b38741cbb1179265d0314919610bca41ae75991cedc67c17d4ad3c6f9d7bbbc6' (1/1)

Locating image lock file images...
The bundle repo (projects.packages.broadcom.com/vsphere/iaas/lci-service/9.0.0-beta/lci-service) is hosting every image specified in the bundle's Images Lock file (.imgpkg/images.yml)


Succeeded
lci-package/.imgpkg/images.yml:  image: projects.packages.broadcom.com/vsphere/iaas/lci-service/9.0.0-beta/lci-service@sha256:e5278d4f538870b707e11ea941b357ddc8c9a393e35bfed320ab940e90d80582
lci-package/config/vkm-svs-bundle.yml:        - image: localhost:5000/vkm-svs/vkm-ui:9.0.0-8594cb6b