Update tanzu-standard package repository after a cluster has been detached from TMC

Products

• VMware Tanzu Mission Control
• VMware Tanzu Kubernetes Grid Management
• VMware vSphere Kubernetes Service

Issue/Introduction

If a cluster (TKGm/TKGs) is detached from Tanzu Mission Control (TMC) and had packages managed by TMC, after the detachment those packages will continue having a reference to the TMC packages repository.

Packages and pods will keep reconciling and working fine, but if there's a change in the TMC standard repository and it becomes unavailable, for example due to an FQDN change, new pods won't be able to pull the images from the unavailable repository.

To avoid this or to recover from this situation, you can manually update the tanzu-standard repository URL to point to a non-TMC dependent one after the cluster is detached.

Note: once the cluster is detached, packages need to be actively managed by the user as they won't be managed by TMC anymore. If any action needs to be performed on the packages, for example updating their versions or the repository, this will need to be done manually.

Reference for tanzu CLI package commands: https://techdocs.broadcom.com/us/en/vmware-tanzu/cli/tanzu-cli/1-5/cli/tanzu-package.html 

Environment

TKGm/TKGs detached from TMC

Resolution

1. Manually update the tanzu-standard repository with an available URL. A reference to updated tanzu-standard repo URLs can be found in Tanzu CLI Package Commands.
   
   # tanzu package repository update tanzu-standard --url projects.packages.broadcom.com/tkg/packages/standard/repo:v2024.4.12 --namespace <tanzu-standard-repo-namespace>
   
   
2. Restart the kapp-controller. Check the namespace where kapp-controller is running with "kubectl get deploy -A | grep kapp-controller":
   
   # kubectl rollout restart -n <kapp-controller-namespace> deploy kapp-controller
   
   
3. Check the URL has been updated:
   
   • # tanzu package repository get tanzu-standard -n <tanzu-standard-repo-namespace>
   • Check all App and PKGI objects are successfully reconciled:
     # kubectl get app,pkgi -A
   • Check the image has been updated with the new repo's URL:
     # kubectl get app -A -oyaml
     
     For example:
     
       spec:
    deploy:
    - kapp:
        rawOptions:
        - --wait-timeout=5m
        - --kube-api-qps=20
        - --kube-api-burst=30
    fetch:
    - imgpkgBundle:
        image: projects.packages.broadcom.com/tkg/packages/standard/repo:v2024.4.12@sha256:<sha>
     
     
4. If the PackageInstall (PKGI) objects fail to reconcile with error "Reconcile failed: Expected to find at least one version", that means the current package's version is not available in the new repo. 
   Update packages' versions:
   
   • To see what versions are available for a package:
     # tanzu package available get <package-name> -n <tanzu-standard-repo-namespace>
   • To update the package's version:
     # tanzu package installed update <package-name> --version <target-version-from-above-output> -n <namespace-where-package-is-installed>
   • Check all App and PKGI objects are successfully reconciled:
     # kubectl get app,pkgi -A