CCI Supervisor Service deployment fails due to image pull errors when deployed from a private registry

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• CCI Supervisor Service fails to deploy because of pods not starting
• Errors will be displayed based on the failure in connectivity in the pod describe :
  

  Message:          failed to pull images: failed to get images: Image svc-cci-service-domain-<cluster id>/cci-namespace-ui-se-11e493####7d2dce3b7ce07-v62351 has failed. Error: Failed to resolve on node esx##.domain.com. Reason: Http request failed. Code 400: ErrorType(2) failed to do request: Head "https://projects.packages.broadcom.com/v2/vcf_cci_service/cci-namespace-ui-service/manifests/sha256:14f7a357002a5d4935b65f4418b52366bfc76aa96b2cd23226c97af725d82fce": <communication error> : ErrImagePull

  OR 

  failed to get images: Timed out with error: Image svc-cci-service-domain-<cluster id>/cci-namespace-ui-se-11e493275cf####dce3b7ce07-v41020 not ready. State: Created Resolver node:

• The private registry in use could be JFrog's Artifactory or any other private registry

Environment

vSphere with Tanzu 8.x with private registry 

Cause

The CCI Service bundle contains image references that points to external repositories instead of references to images hosted within the same repository.
As a result, these images are not automatically rewritten or pulled correctly by Artifactory during mirroring or deployment.

Resolution

The LCI service serves as an alternative to CCI and includes the fix 

Additional Information

Make sure you have copied images using the below documentation if you are using Harbor 
Deploying Supervisor Services from a Private Container Image Registry


The LCI service has all the references in the same repository / bundle 

# /usr/lib/vmware-wcp/tools/imgpkg pull -b projects.packages.broadcom.com/vsphere/iaas/lci-service/9.0.0-beta/lci-service:9.0.0-8594cb6b  -o ./lci-package ; grep -r image: lci-package/

Pulling bundle 'projects.packages.broadcom.com/vsphere/iaas/lci-service/9.0.0-beta/lci-service@sha256:f2e4438c365714e39b4bb955dfba5b344ed020f7f63a3c89474169a6b6b317fa'
  Extracting layer 'sha256:b38741cbb1179265d0314919610bca41ae75991cedc67c17d4ad3c6f9d7bbbc6' (1/1)

Locating image lock file images...
The bundle repo (projects.packages.broadcom.com/vsphere/iaas/lci-service/9.0.0-beta/lci-service) is hosting every image specified in the bundle's Images Lock file (.imgpkg/images.yml)


Succeeded
lci-package/.imgpkg/images.yml:  image: projects.packages.broadcom.com/vsphere/iaas/lci-service/9.0.0-beta/lci-service@sha256:e5278d4f538870b707e11ea941b357ddc8c9a393e35bfed320ab940e90d80582
lci-package/config/vkm-svs-bundle.yml:        - image: localhost:5000/vkm-svs/vkm-ui:9.0.0-8594cb6b