Deployment of VGPU cluster with TKGs fails with the error: 'Insufficient resources. One or more devices are not available on the host.'

Products

• VMware vCenter Server

Issue/Introduction

Worker nodes are stuck in the provisioning state.

Checking the VM tasks from vCenter shows the following error:
"Insufficient resources. One or more devices are not available on the host."

Environment

• VMware vCenter Server 8.x

• vSphere with Tanzu 8.0

Cause

Creating a custom VM class for NVIDIA vGPU devices without configuring CPU and memory reservations can lead to provisioning failures as the system is unable to guarantee the required resources for the vGPU workloads.

Resolution

Starting from vCenter 8.0 Update 3 and later:

• CPU reservation must be set between 0 and 10 MHz.

• All memory must be fully reserved for VM classes utilizing NVIDIA vGPU devices.

Failure to adhere to these requirements result in provisioning errors or resource allocation failures.

Create a Custom VM Class for NVIDIA vGPU Devices