TKG Service Upgrade stuck after upgrading the package from a private registry

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• The TKG Service was upgraded using a local harbor registry after performing a docker pull and docker push command.
• An error such as below will be displayed in the UI : 
  Reason: ReconcileFailed. Message: kapp: Error: waiting on reconcile packageinstall/cluster-api-provider-wcp (packaging.carvel.dev/v1alpha1) namespace: svc-tkg-domain-c1##0: Finished unsuccessfully (Reconcile failed: (message: vendir: Error: Syncing directory '0': Syncing directory '.' with imgpkgBundle contents: Fetching image: GET https://localhost:5000/v2/capw-bundle/manifests/sha256:810f3f01540e6f5ce0e1d68bff937#########108a23a4a7a: MANIFEST_UNKNOWN: manifest unknown; map[Name:capw-bundle Revision:sha256:810f3f01540e6f5ce0e1d68bff937#########108a23a4a7a] )).
  
  

Environment

VMware vSphere Kubernetes Service

Cause

This can occur when packages are bundled using Carvel due to which the pull and push commands will not be able to copy the entire package with all reference images to the private harbor registry.

Resolution

1. Go to " Supervisor > Configure > Supervisor Services > Overview > Kubernetes Service > Manage > Installed Version > Next " , enter the below text under data values and complete the operation to add the correct image URL :
   
   namespace: svc-tkg-domain-c1##0  
template:  
  spec:  
    fetch:  
    - imgpkgBundle:  
        image: projects.packages.broadcom.com/vsphere/iaas/tkg-service/3.#.#/tkg-service:3.#.#

2. Thereafter, pause the PKGI using :
   
   kubectl patch packageinstall  svc-tkg.vsphere.vmware.com -n vmware-system-supervisor-services  --type=merge -p '{"spec": {"paused": true}}'
   
   

3. Edit the package with the right reference using command k edit package command and entering projects.packages.broadcom.com/vsphere/iaas/tkg-service/3.3.3/tkg-service:3.#.#


   1. Run k edit package -n vmware-system-supervisor-services   tkg.vsphere.vmware.com.3.#.#  to edit the package 
   2. Identify the field for image under imgpkgBundle  and enter the correct reference. 

4. Resume the PKGI using command :
   kubectl patch packageinstall  svc-tkg.vsphere.vmware.com -n vmware-system-supervisor-services  --type=merge -p '{"spec": {"paused": false}}'
   
   
   Note : Change the version reference in the above steps according to the version in use . 

Additional Information

Refer article for relocating Supervisor Services to a Private Registry