Package re-conciliation issues in legacy clusters because the repositories are missing.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

1. The kapp-controller pod inside the Guest Cluster is unhealthy and deleting the same doesn't bring it back to a healthy state. 
2. When running "kubectl get pkgi -A", the packages status shows "Reconcile failed: Package <name of the package> not found".
3. On running the command k get packagerepository -A the repositories show missing as below.

k get packagerepository -A

NAMESPACE                   NAME             AGE    DESCRIPTION

Cause

With a cluster upgrade/update, the repositories might disappear. In an ideal scenario, you see the repositories as below.

kubectl get packagerepository -A

NAMESPACE                   NAME             AGE    DESCRIPTION

tanzu-package-repo-global   tanzupackages    359d   Reconcile succeeded

tkg-system                  tanzu-standard   34d    Reconcile succeeded

Resolution

The permanent fix is to upgrade the cluster to a non-legacy TKR version.

Workaround:

A. For a guest cluster with supervisor.

1. Create a packagerepo.yaml specification file that references the target repository version. For example see below

apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageRepository
metadata:
  name: tanzu-standard
  namespace: tkg-system
spec:
  fetch:
    imgpkgBundle:
      image: projects.registry.vmware.com/tkg/packages/standard/repo:v2.2.0_update.2

2. Install the package repository on the TKG cluster as below.

kubectl apply -f packagerepo.yaml

 

 

B. For a standalone management cluster, run the tanzu package repository add command.

tanzu package repository add REPOSITORY-NAME --url REPOSITORY-URL -n REPOSITORY-NAMESPACE. Here REPOSITORY-NAME is a name you choose for the package repository. and REPOSITORY-URL is the OCI registry URL of the package repository.

 

For example, to add the standard package repository, which contains CLI-managed packages included in Tanzu Kubernetes Grid, run the command below. The target --namespace for the standard package repository is tkg-system.

tanzu package repository add tanzu-standard --url projects.registry.vmware.com/tkg/packages/standard/repo:v2024.4.12 --namespace tkg-system

 

 

Post this, the "kubectl get pkgi -A" command output should be as shown in example below.

  NAME                                    PACKAGE-NAME                                 PACKAGE-VERSION                       STATUS               NAMESPACE

  cert-manager                            cert-manager.tanzu.vmware.com                1.7.2+vmware.1-tkg.1                  Reconcile succeeded  tanzu-packages

  contour                                 contour.tanzu.vmware.com                     1.20.2+vmware.2-tkg.1                 Reconciling          tanzu-packages

  fluent-bit                              fluent-bit.tanzu.vmware.com                  1.8.15+vmware.1-tkg.1                 Reconcile succeeded  tanzu-packages

Additional Information

For more information, refer the Broadcom tech docs here- https://techdocs.broadcom.com/#add-repository