Cannot uninstall a package installed with the old Tanzu CLI

Products

• VMware Tanzu Mission Control
• VMware Tanzu Kubernetes Grid Service (TKGs)
• Tanzu Kubernetes Grid

Issue/Introduction

• For TKG clusters that have packages installed via the Tanzu CLI on 1.3.x through 1.6 versions that were subsequently attached to TMC, there might be a conflict in package ownership.
• This will prevent users from managing the packages via TMC and might lead to problems with removal of the original packages.

Cause

The KAPP controller installed via Tanzu CLI for the CLI managed packages is not the same as the KAPP controller installed via TMC. These two KAPP controllers can conflict with each other. Additionally, some of the original packages, such as cert-manager were installed with the package install, app, and all deployments/pods/services in the same namespace. When removing the older cert-manager packages, the service account is removed with the package, but is required to remove the app, leading to a deletion loop that requires manual intervention. Several actions are required to remove the old CLI managed packages and to install the packages again via TMC if desired.

 

 

Resolution

CAUTION: Moving package management from the manually installed KAPP controller to the TMC managed KAPP controller requires removal of the original packages. This might lead to data loss depending on the package in use. The workaround detailed below applies to ONLY contour and cert-manager packages, and doesn't include packages like Harbor, which contain customer data.

 

==============================================================================================

BACKUP:

 

1. 1. BACKUP TLS CERTIFICATE DELEGATION:
      
      # kubectl  -n <NAMESPACE> get tlscertificatedelegations -o yaml > tlscertificatedelegations.yaml
      
      
   2. BACKUP INGRESS SERVICE EXTERNAL IP:
      
      # kubectl -n tanzu-system-ingress get svc
      
      
   3. BACKUP HTTPPROXY IF PRESENT:
      
      # kubectl get httpproxy -A -o yaml > httpproxy.yaml
      
      
   4. ENSURE YOU HAVE A BACKUP OF THE OVERLAY USED TO APPLY THE PACKAGES INITIALLYALONG WITH ANY OTHER CUSTOMIZATIONS.

 

==============================================================================================

REMOVE OLD PACKAGE INSTALLATION:

 

STAGE 1: PACKAGE REPO AND KAPP CONTROLLER:

 

1. 1. 1. Ensure the KAPP controller in the tkg-system namespace is running:
         • Scale down the kapp-controller deployment in tanzu-system namespace (if it is running)
           
           # kubectl scale deployment -n tanzu-system kapp-controller --replicas=0
            
         • NOTE: The kapp controller in tkg-system is the controller installed by Tanzu CLI. The kapp controller installed by TMC will reside in the tanzu-system namespace. We need to get the old kapp controller running if the cluster has already been attached to TMC in order to clean up the apps deployed via Tanzu CLI, which might require the tanzu-system kapp controller to be scaled down. If the cluster hasn't yet been attached to TMC, this step won't be necessary.
           
           
      2. Find and remove the OLD packagerepository (the package repo installed by TMC will be named tanzu-standard, keep this one):
         
         # kubectl get packagerepository -A
         
         # kubectl delete packagerepository -n tanzu-package-repo-global tanzurepo

 

STAGE 2: CONTOUR PACKAGE INSTALL REMOVAL

 

1. 1. 1. Get the package install in preparation for removal:
         
         # kubectl get pkgi -A
         
         
         
      2. Delete the contour PKGI first:
         
         First, back up the service IP:
         
         # kubectl get svc -n tanzu-system-ingress
         
         
         Next, delete contour (in this instance the namespace contour is deployed under is named contour-namespace)
         
         # kubectl delete pkgi -n contour-namespace contour
         
         
         
      3. Ensure all objects are removed from tanzu-system-ingress namespace:
         
         # kubectl get all -n tanzu-system-ingress
         
         
         
      4. Ensure pkgi and app are removed:
         
         # kubectl get pkgi,app -A
         
         
         
      5. Ensure tanzu-system-ingress namespace is removed:
         
         # kubectl get ns
         
         
         NOTE: After deletion of the tanzu-system-ingress namespace, the contour package deletion may hang if the cluster is attached to TMC and the tanzu-system kapp controller is reconciled back into a running state. Scaling down the tanzu-system kapp-controller and restarting the tkg-system kapp-controller should allow the apps and pkgi's for contour to fully delete.
         
         
         
      6. Delete contour-ns namespace for final cleanup:
         
         # kubectl delete ns contour-ns
         
         
         
      7. If the pkgi and app don't delete because of namespace deletion failure for tanzu-system-ingress with errors about the "v1alpha1.data.packaging.carvel.dev" apiservice, we will need to manually delete this apiservice
         
         # kubectl get apiservice | grep packaging
         
         
         
      8. If the resulting package shows state as "False (MissingEndpoints)", delete the apiservice
         
         # kubectl delete apiservice v1alpha1.data.packaging.carvel.dev
         
         
         NOTE FOR PACKAGE REINSTALL: the v1alpha1.data.packaging.carvel.dev apiservice might also cause failures in creation of the NEW contour service if the cluster is already attached to TMC. Errors will appear in the tanzu-system kapp-controller pod log like: "waiting for API service to become ready. Check the status by running 'kubectl get apiservices v1alpha1.data.packaging.carvel.dev -o yaml'"
         
         
         

STAGE 3: CERT MANAGER PACKAGE REMOVAL

 

1. 1. 1. Delete cert-manager PKGI:
         
         # kubectl delete pkgi -n cert-manager cert-manager
         
         
         
      2. Ensure all objects are removed from cert-manager namespace:
         
         # kubectl get all -n cert-manager
         
         
         
      3. ensure pkgi and app are removed:
         
         # kubectl get pkgi,app -A
         
         
         
      4. Ensure cert-manager namespace is removed:
         
         # kubectl get ns
         
         
         
      5. If the app is not deleting, check for warning about namespace removal failing due to ServiceAccount not found. If this error appears, edit the app and remove the finalizer. ENSURE ALL RESOURCES ARE REMOVED FROM CERT-MANAGER NAMESPACE FIRST.

STAGE 4: REMOVE TANZU CLI INSTALLED KAPP CONTROLLER

 

1. 1. 1. Delete the kapp controller in tkg-system as well as the namespace itself:
         
         # kubectl delete deployment kapp-controller -n tkg-system
         
         # kubectl delete svc <kapp-svc> -n tkg-system
         
         # kubectl delete ns tkg-system

 

 

==============================================================================================

REINSTALL PACKAGES VIA TMC

 

1. First, ensure the cluster is attached to TMC.
   
   
2. Ensure KAPP controller in tanzu-system namespace is running.
   
   
   • NOTE: It might be necessary to restart the sync-agent and package-deployment pods in the vmware-system-tmc namespace if the kapp-controller in tanzu-system namespace doesn't exist. This will only be necessary if the cluster was already added to prior to old package removal.
      
3. After the old KAPP is removed, the new packages can be installed from TMC GUI.
   
   
   • If the new Contour Package doesn't deploy, and the kapp controller shows errors: "waiting for API service to become ready. Check the status by running 'kubectl get apiservices v1alpha1.data.packaging.carvel.dev -o yaml'" we need to delete the v1alpha1.data.packaging.carvel.dev apiservice.
     
     # kubectl delete apiservice v1alpha1.data.packaging.carvel.dev