Unable to deploy a cluster on Cilium CNI

Products

VMware vSphere Kubernetes Service

Issue/Introduction

In a vSphere Supervisor environment, a new VKS cluster deploys on Antrea CNI instead of cilium CNI.

When describing the newly deployed VKS cluster, the bootstrapAddons section is missing:

kubectl describe cluster -n <namespace> <cluster name>

From the newly deployed VKS cluster's context, the only CNI pods present are antrea.

kubectl get pods -n kube-system

antrea-agent-<id>
antrea-controller-<id>

VKS clusters using cilium have the following pods in the kube-system namespace:

cilium-<id>
cilium-envoy-<id>
cilium-operator-<id>-<id>

If Hubble was enabled, the below pods are expected:

hubble-relay-<id>-<id>
hubble-ui-<id>-<id>

Environment

vSphere Supervisor

VKS Cluster

Cause

A step was missed or there is misconfiguration in the YAML used to deploy the VKS cluster.

Multiple spec.topology.variables sections* will result in the system removing any additional variables section when deploying the VKS cluster.

This can result in the bootstrapAddons section disappearing when describing the VKS cluster. Without a CNI specified, the system defaults to Antrea CNI.

*Please note that this is specifically spec.topology.variables. This does not apply to variable overrides within a node-pool.

Resolution

Creating a VKS cluster on cilium has the following pre-requisites:

VKS Supervisor service v3.6 or higher installed on the Supervisor Cluster
The VKS cluster YAML must be set to use VKR v1.35 or higher
An addonRepository using version 3.6.0-20260320 minimum installed in the Supervisor Cluster:
```
kubectl get addonrepository -A
```
Older addonRepositories do not have cilium available as an addon.
The latest versions of the addonRepository can be downloaded from the Broadcom Support Portal under vSphere Kubernetes Service -> Drivers & Tools -> VKS Standard Packages.

Connect into the Supervisor cluster context
Confirm that the AddonRepositoryInstall referencing the correct version as per above is in ReconcileSucceeded state:
```
kubectl get addonrepositoryinstall -A

kubectl describe addonrepositoryinstall -n vmware-system-vks-public <name>
```
Check that the cilium addon is listed:
```
kubectl get addons -A | grep cilium
```
Ensure that your cluster YAML contains the following section specifying cilium under spec.topology.variables:
```
variables:
  - name: bootstrapAddons
     value:
        cniRef:
           name: cilium
```
We recommend the variables section to be configured at the bottom of the YAML.

The following warning can be ignored. This does not have impact on the successful deployment of a VKS cluster using cilium.

Warning: addon vmware-system-vks-public/cilium does not have the required CNI category label, it may cause the cluster to be in an NotReady state