Slow vGPU network speed and missing NVLink connections in VCF Private AI Services

Products

VCF Private AI Services

Issue/Introduction

You may observe slow network speeds between vGPUs, and NVLink connections do not display as expected in the GPU topology.

When you run the topology command nvidia-smi topo -m, the output shows SYS or X instead of the expected NV# links, indicating that the GPUs are communicating over the system bus rather than NVLink:

        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      SYS     SYS     SYS     SYS     SYS     SYS     SYS                             N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18                            N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18                            N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18                            N/A
GPU4    SYS     SYS     SYS     SYS      X      SYS     SYS     SYS                             N/A
GPU5    SYS     SYS     SYS     SYS     SYS      X      SYS     SYS                             N/A
GPU6    SYS     SYS     SYS     SYS     SYS     SYS      X      SYS                             N/A
GPU7    SYS     SYS     SYS     SYS     SYS     SYS     SYS      X                              N/A

Additionally, the virtual machine kern.log file continuously reports the following error message

 kernel: [655405.077792] NVRM: subdeviceCtrlCmdPerfGetVideoEnginePerfmonSample_VF: Call not supported with SMC Enabled

Querying the GPU mode status with nvidia-smi --query-gpu=index,name,mig.mode.current --format=csv confirms that Multi-Instance GPU (MIG) mode is currently enabled on the affected devices.

index, name, mig.mode.current
0, NVIDIA H200, Enabled
1, NVIDIA H200, Enabled
2, NVIDIA H200, Enabled
3, NVIDIA H200, Enabled
4, NVIDIA H200, Disabled
5, NVIDIA H200, Disabled
6, NVIDIA H200, Enabled
7, NVIDIA H200, Disabled

Environment

VCF Private AI Services
NVIDIA H200 GPUs (HGX H100/H200)
NVIDIA vGPU on NVSwitch systems

Cause

This issue occurs because MIG mode is configured on the GPUs instead of time-slicing. Only time-sliced vGPUs support peer-to-peer (P2P) communication; MIG-backed vGPUs are not supported over NVLink or otherwise.

On NVSwitch systems, only full-framebuffer time-sliced profiles participate in the NVSwitch fabric. For example, on the H200 SXM5 141GB, the time-sliced H200X-141C profile participates in the NVSwitch fabric, whereas the MIG-backed H200X-7-141C profile does not.

Resolution

To resolve this issue, you must disable MIG mode and configure time-slicing for the GPUs. The steps depend on how your GPUs are provisioned:

For vGPU VMs:

Create a new VM class with updated GPU profiles that have time-slicing configured.
Power off all vGPU VMs currently running with MIG profiles.
Power on the vGPU VMs using the newly created time-slice profiles. The H200 devices automatically switch to time-sliced mode based on the applied vGPU VM profile.

For Passthrough GPUs:

Delete all MIG instances within the passthrough VMs.
Invoke the command nvidia-smi -mig 0 within the VMs to disable MIG mode.

Once the switch to time-sliced vGPUs is complete, the Distributed Lock Manager (DLM) enables NVLink. Note that all P2P vGPUs must reside in the same VM for the NVSwitch fabric to function correctly.

Additional Information

https://docs.nvidia.com/ai-enterprise/release-8/latest/infra-software/vgpu/features/p2p.html
https://techdocs.broadcom.com/us/en/vmware-cis/private-ai/foundation-with-nvidia/9-0/private-ai-foundation-9-x/deploying-private-ai-foundation-with-nvidia/configure-vgpu-based-vm-classes-for-ai-workloads.html