Model endpoints fail to initialize because the `gpu-operator` cannot deploy NVIDIA driver pods

Products

• VCF Private AI Services

Issue/Introduction

1."gpu-operator" pods fail to deploy the NVIDIA driver component.

2.The PAIS environment utilized the default Ubuntu template v1.33.6.

3.Logs from the pod nvidia-driver-daemonset-xxx in  namespace gpu-operator  showed pre-check failures when attempting to fetch kernel-specific NVIDIA drivers.

"

starting installation of NVIDIA driver version 570.158.01-grid for Linux kernel version 6.8.0-90-generic

Resolving Linux kernel version...

Could not resolve Linux kernel version

unloading NVIDIA driver kernel version

Unmounting NVIDIA driver rootfs... "

"

Environment

VCF Private AI Services 2.1

Cause

The upstream Ubuntu repository removed the specific Linux kernel version (6.8.0-90-generic) used in the v1.33.6 template, preventing the `gpu-operator` from retrieving required headers for driver compilation.

Resolution

Please find the steps below to upgrade the PAIF cluster to vkr 1.34.8:

 

1. Download vkr 1.34.8 template from below link:

https://wp-content.vmware.com/v2/latest/

 

Then upload it to the content library of the [PAIF namespace] namespace via vSphere UI.

 

2. Verify the upload completely and record the specific version string (e.g., "v1.34.8+vmware.1-figs-vkr.1") by running:

kubectl get kubernetesreleases.kubernetes.vmware.com

 

3. Backup the current configuration:

kubectl -n [PAIF namespace] get pcfg default -o yaml > ./default_pcfg_backup.yaml

 

4. Edit the configuration:

kubectl -n [PAIF namespace] edit pcfg default

 

5. Update the vksControlPlane section by adding the upgradeStrategy as shown below:

 

Before:

 

vksControlPlane:

    virtualMachineClassName: vm_class_VKS_control_plane_node

    storageClassName: storage_policy_VKS_control_plane_node

 

After:

 

vksControlPlane:

    virtualMachineClassName: vm_class_VKS_control_plane_node

    storageClassName: storage_policy_VKS_control_plane_node

        upgradeStrategy:

            manual:

                version: [Version from Step 2]

 

6. Save and exit (:wq) to ensure the file is edited successfully.

 

7. Monitor the upgrade status to confirm that both the Control Plane (CP) and worker nodes become available:

kubectl -n [PAIF namespace] get cluster

 

8. Once complete, all pods in the gpu-operator namespace should reconcile successfully.