Mitigating Gateway API CRD collisions between VKS Guest Cluster bootstrap package and third-party software installations

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

VKS includes Gateway API CRDs by default as one of the built-in VKr components included in Guest Clusters.

Some third-party software bundles (Helm charts, etc.) may also bundle the Gateway API CRDs, causing a conflict in ownership
This particularly can cause issues when attempting to install a different version of the CRDs than is provided by the VKr for your Cluster.

Gateway API CRDs are owned by the gateway-api Carvel Package, which will continually reconcile them to the version provided by the VKr.
This can cause the installation of third-party software to fail, or limit functionality if attempting to use features present in newer releases of the Gateway API "standard" channel CRDs than are present in your Cluster.

Environment

VKS 3.0.0+, VKr 1.27.16+

Resolution

Many third-party software bundles provide options to omit installing CRDs and this can be used to allow VKS to manage Gateway API CRDs and not conflict with another installation.
This is particularly popular in Helm charts, to enable managing CRDs outside of Helm.

For example Traefik provides a separate Helm chart to manage CRDs, and allows users to skip installing the Gateway API CRDs, see: traefik-helm-chart 

Another option to resolve this conflict is to opt out of VKS installing and managing Gateway API CRDs using a custom ClusterBootstrap resource.

Setting the "tkg.tanzu.vmware.com/custom-clusterbootstrap" annotation on the ClusterBootstrap resource associated with your Cluster and removing the "gateway-api.*" package from the spec.additionalPackages list.
This will allow installation of any desired Gateway API CRD version, channel, etc. without conflict with VKS-provided CRDs.