PAIS Model Endpoints Become Unresponsive After Initial Requests: Verifying NVIDIA vGPU Licensing

Products

• VCF Private AI Services

Issue/Introduction

A model deployed on PAIS responds to the first few completions normally, then becomes nearly non-responsive, with requests eventually getting aborted reporting the below error:

[engine.py:285] Aborted request chatcmpl-...

 

Environment

Private AI Services (PAIS) 2.0

Private AI Services (PAIS) 2.1

VMware Cloud Foundation 9.0

VMware Cloud Foundation 9.1

Cause

This is commonly caused by the GPU node failing to acquire a license from your NVIDIA vGPU License Server — the license pool is exhausted, or no licenses are available.

Without a license, the vGPU falls back to a restricted/degraded mode.

Resolution

Prerequisites:

• kubectl access to the Supervisor namespace where your PAIS instance is deployed.
• base64 utility available locally (bundled with most Operating Systems).

Step 1: Get the kubeconfig for the PAIS VKS cluster

PAIS deploys your models onto a dedicated VKS (vSphere Kubernetes Service) guest cluster.
Its kubeconfig is stored as a Kubernetes secret in your Supervisor namespace, named -kubeconfig.

With KUBECONFIG pointing at your Supervisor namespace, run:

kubectl get secret $(kubectl get secret | grep kubeconfig | awk '{print $1}') -o jsonpath='{.data.value}' | base64 -d > vks-kubeconfig.yml

This writes the VKS cluster's kubeconfig to vks-kubeconfig.yml. Point KUBECONFIG at that file for the remaining commands:

export KUBECONFIG=./vks-kubeconfig.yml

 

Step 2: Check the vGPU license status on the GPU node

Find the nvidia-device-plugin-daemonset-* pod running in the gpu-operator namespace on the node hosting your model:

kubectl -n gpu-operator get pods -o wide

Exec into that pod and check the license status with nvidia-smi:

kubectl -n gpu-operator exec -it -c nvidia-device-plugin -- nvidia-smi -q | grep -i license

Interpret the output:

Licensed — the GPU has successfully acquired a license and should perform normally:

vGPU Software Licensed Product
License Status : Licensed (Expiry: YYYY-MM-DD HH:MM:SS GMT)

Unlicensed (Restricted) — the GPU has not acquired a license, which will cause degraded performance and, eventually, unresponsive models:

vGPU Software Licensed Product
License Status : Unlicensed (Restricted)

If the GPU is unlicensed, you must acquire proper licensing for the models to operate.