vllm worker fails with "No CUDA GPUs are available"
Article ID: 448785
Updated On:
Products
VMware Tanzu Platform Core
Issue/Introduction
AI Services tile installed with On Platform vLLM model.
The vllm-worker process fails to initialise the vLLM engine with error ""RuntimeError: No CUDA GPUs are available"
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] EngineCore failed to start.^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] Traceback (most recent call last):^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] File "/var/vcap/packages/vllm/.python/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 1082, in run_engine_core^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] File "/var/vcap/packages/vllm/.python/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] return func(*args, **kwargs)^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] File "/var/vcap/packages/vllm/.python/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 848, in __init__^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] super().__init__(^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] File "/var/vcap/packages/vllm/.python/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 114, in __init__^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] self.model_executor = executor_class(vllm_config)^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] File "/var/vcap/packages/vllm/.python/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] return func(*args, **kwargs)^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] File "/var/vcap/packages/vllm/.python/lib/python3.11/site-packages/vllm/v1/executor/abstract.py", line 103, in __init__^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] self._init_executor()^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] File "/var/vcap/packages/vllm/.python/lib/python3.11/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] self.driver_worker.init_device()^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] File "/var/vcap/packages/vllm/.python/lib/python3.11/site-packages/vllm/v1/worker/worker_base.py", line 312, in init_device^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] self.worker.init_device() # type: ignore^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] File "/var/vcap/packages/vllm/.python/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] return func(*args, **kwargs)^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] File "/var/vcap/packages/vllm/.python/lib/python3.11/site-packages/vllm/v1/worker/gpu_worker.py", line 255, in init_device^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] torch.accelerator.set_device_index(self.device)^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] File "/var/vcap/packages/vllm/.python/lib/python3.11/site-packages/torch/accelerator/__init__.py", line 190, in set_device_index^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] torch._C._accelerator_setDeviceIndex(device_index)^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] File "/var/vcap/packages/vllm/.python/lib/python3.11/site-packages/torch/cuda/__init__.py", line 424, in _lazy_init^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] torch._C._cuda_init()^M
(EngineCore pid=546584) ERROR 07-14 22:24:45 [core.py:1108] RuntimeError: No CUDA GPUs are availableEnvironment
AI Services Tile with On Platform vLLM Model
PCI Passthrough
Cause
Output of "nvtop" and "nvidia-smi" confirm that the Worker VM has visibility of the GPU.
However this also confirms that MiG is enabled and that should not be the case with PCI Passthrough, see Configure NVIDIA vGPU or GPU Passthrough for AI Workloads on the ESX Hosts
# nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.211.01 Driver Version: 570.211.01 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A100-SXM4-80GB On | 00000000:0B:00.0 Off | On |
| N/A 22C P0 49W / 500W | 0MiB / 81920MiB | N/A Default |
| | | Enabled |
+-----------------------------------------+------------------------+----------------------+
| MIG devices: |
+------------------+----------------------------------+-----------+-----------------------+
| GPU GI CI MIG | Memory-Usage | Vol| Shared |
| ID ID Dev | BAR1-Usage | SM Unc| CE ENC DEC OFA JPG |
| | | ECC| |
|==================+==================================+===========+=======================|
| No MIG devices found |
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+ Resolution
Disable MiG and power off/on the VM.
nvidia-smi -i <GPU_ID> -mig 0Run Apply Changes on AI Service Tile to complete the installation.
Additional Information
After disabling MIG
# nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.211.01 Driver Version: 570.211.01 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A100-SXM4-80GB On | 00000000:0B:00.0 Off | On |
| N/A 22C P0 49W / 500W | 0MiB / 81920MiB | N/A Default |
| | | Enabled |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 1832 C VLLM::EngineCore 69618MiB |
+-----------------------------------------------------------------------------------------+ Feedback
Yes
No
Powered by
