TAP installed on a TMC managed cluster upgrade failed due to fluxcd-source-controller package install reconcile failed

Products

• VMware Tanzu Application Platform

Issue/Introduction

The customer is using TAP which was installed on a TMC managed Kubernetes Cluster. They followed the doc: Upgrade Tanzu Application Platform to upgrade TAP version with command "tanzu package installed update tap -p tap.tanzu.vmware.com -v ${TAP_VERSION}  --values-file tap-values.yaml -n tap-install", but it failed with below error:

6:44:44PM: Fetch succeeded
6:44:44PM: Template succeeded
Error: packageinstall/tap (packaging.carvel.dev/vlalpha1) namespace: tap-install: Reconciling: kapp: Error: waiting on reconcile packageinstall/fluxcd-source-controller (packaging.carvel.dev/vlalpha1) namespace: tap-install:
Finished unsuccessfully (Reconcile failed: (message: kapp: Error: Ownership errors:
- Resource 'clusterrolebinding/fluxcd-source-manager-rolebinding (rbac.authorization.k8s.io/v1) cluster' is already associated with a different label 'kapp.k14s.io/app=######'
- Resource 'customresourcedefinition/example.source.toolkit.fluxcd.io (apiextensions.k8s.io/v1) cluster' is already associated with a different label 'kapp.k14s.io/app=######'
- Resource 'clusterrole/fluxcd-source-manager-role (rbac.authorization.k8s.io/v1) cluster' is already associated with a different label 'kapp.k14s.io/app=######'
- Resource 'customresourcedefinition/example.source.toolkit.fluxcd.io (apiextensions.k8s.io/v1) cluster' is already associated with a different label 'kapp.k14s.io/app=######")). Reconcile failed: Error (see .status.usefulErrorMessage for details)

$ tanzu package installed list -A

  tap-install                   fluxcd-source-controller            fluxcd.source.controller.tanzu.vmware.com            0.36.1+tanzu.2         Reconcile failed
  tap-install                   scanning                            scanning.apps.tanzu.vmware.com                       1.7.9                  Reconcile failed
  tap-install                   tap                                 tap.tanzu.vmware.com                                 1.7.9                  Reconcile failed

Cause

TAP and TMC both try to install a Flux source controller. TMC has logic that detects a present source controller, and skips install if one is present. However, TAP does not.

Resolution

There are two ways to workaround this:

1. Add the following line to TAP values to skip the source controller install:

• Below TAP 1.10 source-controller package metadata name as following:

excluded_packages:
  -  fluxcd.source.controller.tanzu.vmware.com 

• From TAP 1.10 onwards source-controller package metadata name is updated to following:

excluded_packages:
  - fluxcd-source-controller.tanzu.vmware.com

2. Disable the TMC helm service, then install TAP, then add the TMC helm service back.

Based on TMC doc Install Tanzu Application Platform on a Cluster - Prerequisites,
Because TAP uses an earlier version of FluxCD, and is therefore incompatible with the deployed version, you must first disable CD/Helm on the cluster before attempting to install TAP.

For disabling Helm Service, can refer to TMC doc Disable Helm Service.