Nvidia GPU operator fails to install on TKGi

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

GPU operator is being installed on TKGi cluster as per the Install GPU Kubernetes Operator with:

helm install --wait --generate-name \
-n gpu-operator --create-namespace \
nvidia/gpu-operator \
  --set driver.enabled=true \
  --set toolkit.enabled=true \
  --set toolkit.env[0].name=CONTAINERD_CONFIG \
  --set toolkit.env[0].value=/var/vcap/jobs/containerd/config/config.toml \
  --set toolkit.env[1].name=CONTAINERD_SOCKET \
  --set toolkit.env[1].value=/var/vcap/sys/run/containerd/containerd.sock \
  --set toolkit.env[2].name=CONTAINERD_RUNTIME_CLASS \
  --set toolkit.env[2].value=nvidia \
  --set toolkit.env[3].name=CONTAINERD_SET_AS_DEFAULT \
  --set-string toolkit.env[3].value="true"

But it fails to install with error:

.spec.toolkit.env[3].value_type: field not declared in schema

Environment

TKGi GPU cluster

Cause

Depending on the gpu operator version, different environment variable are supported.

In this example, CONTAINERD_SET_AS_DEFAULT is no longer supported in gpu operator v26.3

Resolution

Refer to the GPU Operator documentation and the supported environment variables, 26.3 Configuration options

Install gpu operator with only supported parameters and env variable, eg for 26.3 and in order to use the older method of deployment disable CDI method:

cdi:
  enabled: false
  nriPluginEnabled: false

toolkit:
  enabled: true
  env:
    # TKGi BOSH-managed containerd config and socket paths
    - name: CONTAINERD_CONFIG
      value: /var/vcap/jobs/containerd/config/config.toml
    - name: CONTAINERD_SOCKET
      value: /var/vcap/sys/run/containerd/containerd.sock
    # create the nvidia RuntimeClass Kubernetes object
    - name: CONTAINERD_SET_AMD64_RUNTIME_CLASS
      value: "true"

 

Deploy the Operator where gpu-operator-values.yaml contains above config:

helm install gpu-operator nvidia/gpu-operator   --version v26.3.2   --namespace gpu-operator   -f gpu-operator-values.yaml

Additional Information

Also note that CDI was disabled by default in v25.3 and earlier but was enabled in 25.10 onwards. This can impact the installation of the Pods.