Migration from using CNR contour to TAP contour
Article ID: 298454
Updated On:
Products
Issue/Introduction
Product: VMware Tanzu Application Platform
Component: Cloud Native Runtimes
Affected versions: TAP v1.3/CNR v2.0
How can I successfully migrate to CNR v2.0?
CNR v2.0 will no longer ship with Contour. This means that CNR will not be usable without a compatible ingress provider (Contour only at this moment) already installed on the cluster.
All customers relying on CNR provided Contour in any form will experience varying levels of impact and downtime when upgrading to CNR 2.0. The various upgrade scenarios and levels of impact are explored further in the Upgrade Scenarios section below.
On the other hand, customers relying on only TAP Contour (the default configuration for TAP) should not experience any problems during this upgrade related to removing CNR Contour.
Because CNR 1.3 introduced a breaking change to mixing TAP and CNR Contour, many customers may have already moved away from relying on CNR Contour.
Environment
Resolution
Upgrade Scenarios
The best way to upgrade is to first switch away from using a CNR created Contour, and then onto a TAP created Contour. How this is done will depend on your current CNR configuration and your tolerance for downtime.
Scenario 1
CNR using TAP Contour for both Internal and External
cnrs: ingress: reuse_crds: true external: namespace: tanzu-system-ingress internal: namespace: tanzu-system-ingress
This is the default configuration in TAP. If you are using CNR from TAP installed with a profile, you can follow the TAP upgrade process as you would normally.
If you have installed CNR 1.3 individually from the TAP package repository, CNR is configured to use TAP Contour, and the CNR 2.0.x package is available, then you can run tanzu package install cloud-native-runtimes -p cnrs.tanzu.vmware.com -v 2.0.0 --values-file <values.yaml> -n tap-install to upgrade the package version.
Scenario 2
CNR using CNR Contour for both Internal and External (no TAP Contour on Cluster)
cnrs: ingress: reuse_crds: false external: namespace: "" internal: namespace: "" excluded_packages: contour.tanzu.vmware.com
In this case, the Contour installed on the cluster comes from CNR. That means, if we delete CNR, or upgrade to a version of CNR without Contour, there will be no ingress provider on the cluster.
The smoothest way to get past this is to switch off of CNR Contour before upgrading.
1. Reconfigure CNR 1.3 from expecting CNR Contour to expecting TAP Contour:
- Create a secret with the overlay to mark all Contour resources in CNR 1.3 with the orphan delete strategy: kubectl -n tap-install create secret generic cnrs-overlay-orphan-contour -o yaml --from-file=cnrs-overlay-orphan-contour.yaml
- Annotate the CNR 1.3 PackageInstall with that overlay: kubectl annotate packageinstall cnrs "ext.packaging.carvel.dev/ytt-paths-from-secret-name.0"="cnrs-overlay-orphan-contour" -n tap-install
- Once it is reconciled, set “paused: true” on the CNR 1.3.0 PackageInstall: kubectl -n tap-install patch pkgi cnrs --type=merge --patch '{"spec":{"paused":true}}'
- Update the CNR 1.3 config values with this ingress block:
ingress: external namespace: tanzu-system-ingress internal namespace: tanzu-system-ingress
- Remove the overlay annotation added in step b.
- Unpause the CNR PackageInstall and let it reconcile. This will orphan the contour resources, and reconfigure CNR to expect Contour in the tanzu-system-ingress namespace.
Note: If workloads are created in this state for some reason, they will not be accessible until Contour is installed in the tanzu-system-ingress namespace. Existing workloads will remain available via the orphaned Contour installation.
2. Install TAP Contour and clean up old CNR Contour
- Install TAP Certmanager (if not installed already) and TAP Contour.
- This will deploy Contour in the tanzu-system-ingress namespace, which CNR was previously configured to expect. All new Knative Services will be served by this Contour.
- Update DNS records to point to new tanzu-system-ingress LB IP.
- Clean up old CNR Contour namespaces and resources:
Kubectl delete namespace contour-external Kubectl delete namespace contour-internal Kubectl delete clusterrolebindings.rbac.authorization.k8s.io knative-contour-external Kubectl delete clusterrolebindings.rbac.authorization.k8s.io knative-contour-internal
3. Upgrade as normal to CNR 2.0
-
Update config values, as reuse_crds value will no longer exist.
Scenario 3
CNR using TAP Contour CRDs but CNR Contour for both Internal and External
cnrs: ingress: reuse_crds: true external: namespace: "" internal: namespace: ""
This scenario is broken as of CNR 1.3 / TAP 1.2. We don't support this configuration.
Getting out of this is a little more straightforward, since a Contour (hopefully a TAP provided one) is already installed. Just reconfigure the values to provide the namespace of the existing Contour instance.
This will delete the CNR created Contour. Then proceed to upgrade like in the first scenario.
Scenario 4
CNR using TAP Contour CRDs, CNR Contour for External, TAP Contour for Internal
cnrs: ingress: reuse_crds: true external: namespace: "" internal: namespace: tanzu-system-ingress
This scenario is broken as of CNR 1.3 / TAP 1.2. We don't support this configuration.
Getting out of this is a little more straightforward, since a Contour (hopefully a TAP provided one) is already installed. Just reconfigure the values to provide the namespace of the existing Contour instance.
This will delete the CNR created Contour. Then proceed to upgrade like in the first scenario.
Scenario 5
CNR using TAP Contour CRDs, TAP Contour for External, CNR Contour for Internal
cnrs: ingress: reuse_crds: true external: namespace: tanzu-system-ingress internal: namespace: ""
This scenario is broken as of CNR 1.3 / TAP 1.2. We don't support this configuration.
Getting out of this is a little more straightforward, since a Contour (hopefully a TAP provided one) is already installed. Just reconfigure the values to provide the namespace of the existing Contour instance.
This will delete the CNR created Contour. Then proceed to upgrade like in the first scenario.
Appendix : An overlay to add orphan delete strategy to Contour resources
#@ load("@ytt:overlay", "overlay")
#@ def add_orphan_annotation():
metadata:
#@overlay/match missing_ok=True
annotations:
#@overlay/match missing_ok=True
kapp.k14s.io/delete-strategy: orphan
#@ end
#! Start with CRDs
#@overlay/match by=overlay.subset({"kind":"CustomResourceDefinition","metadata":{"labels":{"networking.knative.dev/ingress-provider": "contour"}}}), expects=[5,10]
--- #@ add_orphan_annotation()
#! Now do all external stuff
#@overlay/match by=overlay.subset({"kind":"Namespace","metadata":{"name":"contour-external"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"ClusterRoleBinding","metadata":{"name":"knative-contour-external"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"ServiceAccount","metadata":{"name":"contour", "namespace": "contour-external"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"ServiceAccount","metadata":{"name":"envoy", "namespace": "contour-external"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"ServiceAccount","metadata":{"name":"contour-certgen", "namespace": "contour-external"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"RoleBinding","metadata":{"name":"contour", "namespace": "contour-external"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"Role","metadata":{"name":"contour-certgen", "namespace": "contour-external"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"ConfigMap","metadata":{"name":"contour", "namespace": "contour-external"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"Job","metadata":{"name":"contour-certgen-v1.19.1", "namespace": "contour-external"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"Service","metadata":{"name":"contour", "namespace": "contour-external"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"Service","metadata":{"name":"envoy", "namespace": "contour-external"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"Deployment","metadata":{"name":"contour", "namespace": "contour-external"}})
---
metadata:
#@overlay/match missing_ok=True
annotations:
#@overlay/match missing_ok=True
kapp.k14s.io/delete-strategy: orphan
spec:
template:
metadata:
#@overlay/match missing_ok=True
annotations:
#@overlay/match missing_ok=True
kapp.k14s.io/delete-strategy: orphan
#@overlay/match by=overlay.subset({"kind":"DaemonSet","metadata":{"name":"envoy", "namespace": "contour-external"}})
---
metadata:
#@overlay/match missing_ok=True
annotations:
#@overlay/match missing_ok=True
kapp.k14s.io/delete-strategy: orphan
spec:
template:
metadata:
#@overlay/match missing_ok=True
annotations:
#@overlay/match missing_ok=True
kapp.k14s.io/delete-strategy: orphan
#! Now do internal
#@overlay/match by=overlay.subset({"kind":"Namespace","metadata":{"name":"contour-internal"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"ClusterRoleBinding","metadata":{"name":"knative-contour-internal"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"ServiceAccount","metadata":{"name":"contour", "namespace": "contour-internal"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"ServiceAccount","metadata":{"name":"envoy", "namespace": "contour-internal"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"ServiceAccount","metadata":{"name":"contour-certgen", "namespace": "contour-internal"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"RoleBinding","metadata":{"name":"contour", "namespace": "contour-internal"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"Role","metadata":{"name":"contour-certgen", "namespace": "contour-internal"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"ConfigMap","metadata":{"name":"contour", "namespace": "contour-internal"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"Job","metadata":{"name":"contour-certgen-v1.19.1", "namespace": "contour-internal"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"Service","metadata":{"name":"contour", "namespace": "contour-internal"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"Service","metadata":{"name":"envoy", "namespace": "contour-internal"}})
--- #@ add_orphan_annotation()
#@overlay/match by=overlay.subset({"kind":"Deployment","metadata":{"name":"contour", "namespace": "contour-internal"}})
---
metadata:
#@overlay/match missing_ok=True
annotations:
#@overlay/match missing_ok=True
kapp.k14s.io/delete-strategy: orphan
spec:
template:
metadata:
#@overlay/match missing_ok=True
annotations:
#@overlay/match missing_ok=True
kapp.k14s.io/delete-strategy: orphan
#@overlay/match by=overlay.subset({"kind":"DaemonSet","metadata":{"name":"envoy", "namespace": "contour-internal"}})
---
metadata:
#@overlay/match missing_ok=True
annotations:
#@overlay/match missing_ok=True
kapp.k14s.io/delete-strategy: orphan
spec:
template:
metadata:
#@overlay/match missing_ok=True
annotations:
#@overlay/match missing_ok=True
kapp.k14s.io/delete-strategy: orphan
Feedback
