PAIS Model Endpoints Become Unresponsive After Initial Requests: Verifying NVIDIA vGPU Licensing

Products

• VCF Private AI Services

Issue/Introduction

A model deployed on PAIS responds to the first few completions normally, then becomes nearly non-responsive, with requests eventually getting aborted

[engine.py:285] Aborted request chatcmpl-..

Environment

• Private AI Services (PAIS) 2.x
• VCF 9.x

Cause

This is commonly caused by the GPU node failing to acquire a license from the NVIDIA vGPU License Server - the license pool is exhausted, or no licenses are available. Without a license, the vGPU falls back to a restricted/degraded mode. 

Resolution

Prerequisites:

• kubectl access to the Supervisor namespace where the PAIS instance is deployed.
• base64 available locally (bundled with most OSes).

1. Get the kubeconfig for the PAIS VKS cluster PAIS deploys the models onto a dedicated VKS (vSphere Kubernetes Service) guest cluster. Its kubeconfig is stored as a Kubernetes secret in the Supervisor namespace, named -kubeconfig. 
   
   
   1. With KUBECONFIG pointing at the Supervisor namespace, run:
      
      kubectl get secret $(kubectl get secret | grep kubeconfig | awk '{print $1}') -o jsonpath='{.data.value}' | base64 -d > vks-kubeconfig.yml
      
      This writes the VKS cluster's kubeconfig to vks-kubeconfig.yml.
      
      
   2. Point KUBECONFIG at that file for the remaining commands:
      
      export KUBECONFIG=./vks-kubeconfig.yml
      
      
2. Check the vGPU license status on the GPU node Find the nvidia-device-plugin-daemonset-* pod running in the gpu-operator namespace on the node hosting the model:
   
   
   1. kubectl -n gpu-operator get pods -o wide
      
      
   2. Exec into that pod and check the license status with
      
      nvidia-smi: kubectl -n gpu-operator exec -it -c nvidia-device-plugin -- nvidia-smi -q | grep -i license
      
      
3. Interpret the output:
   
   
   1. Licensed — the GPU has successfully acquired a license and should perform normally:
      
      vGPU Software Licensed Product
License Status : Licensed (Expiry: <Date>)
      
      
   2. Unlicensed (Restricted) — the GPU has not acquired a license, which will cause degraded performance and, eventually, unresponsive models:
      
      vGPU Software Licensed Product
License Status : Unlicensed (Restricted)
      
      If the GPU is unlicensed, acquire proper licensing for the models to operate.

Additional Information

Obtaining, generating, or validating NVIDIA NGC API keys and license tokens is managed entirely within the customer's NVIDIA NGC portal account and is outside the scope of Broadcom Support. Please work with your organization's NVIDIA account administrator to acquire valid credentials.