Cilium Operator Fails on VKr 1.36+ Clusters with kube-proxy disabled and Gateway API enabled in VKS 3.7
search cancel

Cilium Operator Fails on VKr 1.36+ Clusters with kube-proxy disabled and Gateway API enabled in VKS 3.7

book

Article ID: 448213

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

When performing a fresh installation or upgrading to VMware Kubernetes release (VKr) 1.36+, the Cilium operator may enter a failed state. This issue is specific to clusters created with Cilium as the Container Network Interface (CNI), where kube-proxy is disabled for the cluster, and gatewayAPI.enabled: true is configured in the Cilium AddonConfig. For any other scenario, the configuration functions completely fine.

Any secondary addons dependent on Gateway API CRDs (such as AKO, Contour, or Istio) will also move to a FAILED state once the workaround is initiated, unless a compatible external Gateway API is installed.

Environment

  • VMware Kubernetes Service (VKS) 3.7.0 and later

  • VMware Kubernetes release (VKr) 1.36 and later

  • Cilium CNI installed via standard packages, version v1.19.4+

  • kube-proxy disabled on the cluster

Cause

VKr (OVA) 1.36+ ships with Gateway API v1.5.1 and automatically installs it unless explicitly opted out. However, Cilium standard-package releases v1.19.4+ are compiled with Gateway API v1.4.0-rc.2.

On a fresh install in VKr 1.36, or during a cluster upgrade to VKr 1.36 (where existing v1.4.1 CRDs are automatically bumped to v1.5.1), this version mismatch occurs. The Cilium operator does not support the v1.5.1 CRD schema, leading to a failure when the specific Cilium configuration (disabled kube-proxy and enabled Gateway API) is applied.

Resolution

To utilize Cilium's Gateway API support on VKr 1.36+ clusters, the VKr-shipped Gateway API addon must be explicitly opted out of, and the compatible Gateway API version must be installed manually.

Step 1: Opt out of the VKr-shipped Gateway API addon

On the Supervisor, a cluster must be provisioned without gateway-api. The opt-out label must be added to the Cluster object using the following command:

kubectl label cluster <cluster-name> -n <cluster-namespace> addon.addons.kubernetes.vmware.com/gateway-api=unmanaged

The Cluster resource will reflect the unmanaged label:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <cluster-name>
  namespace: <cluster-namespace>
  labels:
    addon.addons.kubernetes.vmware.com/gateway-api: unmanaged
spec:
  topology:
    class: tanzukubernetescluster
    version: v1.36.x+vmware.1-vks.1

Note: VKS automatically removes the existing Gateway API v1.5.1 PackageInstall and all Gateway API CRDs. The opt-out label is preserved across cluster upgrades.

To verify the CRDs were removed in the VKS Cluster, run:

kubectl get crds | grep gateway.networking.k8s.io


Step 2: Enable Gateway API support in the Cilium AddonConfig

On the Supervisor, spec.values.gatewayAPI.enabled must be set to true.

Note: kube-proxy needs to be already disabled on the cluster for this to take effect.

apiVersion: addons.kubernetes.vmware.com/v1alpha1
kind: AddonConfig
metadata:
  annotations:
    clusteraddon.addons.kubernetes.vmware.com/owned-for-deletion: "true"
  name: <cluster-name>-cilium
  namespace: <cluster-namespace>
spec:
  values:
    gatewayAPI:
      enabled: true


Step 3: Install the Cilium-compatible Gateway API CRD minor version

In the VKS Cluster, the upstream standard-install.yaml manifest must be applied for the compatible minor version vendored by Cilium v1.19.4+. A compatible patch (e.g., v1.4.1) can be installed as follows:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/standard-install.yaml

Note: These CRDs will not be managed or upgraded by VKS once installed manually. If a future Cilium release vendors a newer Gateway API version, this CRD installation must be manually updated to match.


Step 4: Restart the Cilium operator

In the VKS Cluster, the cilium-operator deployment must be restarted manually:

kubectl rollout restart deployment cilium-operator -n kube-system


Step 5: Create a GatewayClass for Cilium

In the VKS Cluster, the following GatewayClass must be created:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: cilium-gateway
  namespace: kube-system
spec:
  controllerName: io.cilium/gateway-controller
  description: "Cilium Gateway API Controller"

Step 6: Verify the GatewayClass is Accepted

In the VKS Cluster, the GatewayClass must be validated to ensure it is correctly created and ready:

kubectl get gatewayclass -A

Expected Output:

NAME             CONTROLLER                     ACCEPTED   AGE
cilium-gateway   io.cilium/gateway-controller   True       108s

Once ACCEPTED is True, Gateway, HTTPRoute, and other Gateway API resources can be configured, and Cilium will reconcile them against the manually-installed Gateway API CRDs.

Powered by Wolken Software