Private AI Services ModelEndpoints Fail - ValueError: Free memory on device on startup is less than desired GPU memory utilization

Products

• VCF Private AI Services

Issue/Introduction

Newly deployed modelendpoint pods are failing in CrashLoopBackOff state.

While connected to the PAIS cluster context where the modelendpoint pods are deploying, the following symptoms are observed:

• Describing the failing modelendpoint pods shows that the container main is Terminated with an Error repeatedly:
  

  kubectl describe pod -n <modelendpoint namespace> <modelendpoint pod>

   

  
  
  
• Viewing the logs for the main container for the affected modelendpoint pod shows a long stacktrace with an error similar to the following:
  

  kubectl logs -n <modelendpoint namespace> <modelendpoint pod> -c main
  
  ValueError: Free memory on device (<current memory>/<max memory> GiB) on startup is less than desired GPU memory utilization (0.9, <memory> GIB). Decrease GPU memory utilization or reduce GPU memory used by other processes. 
  RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): ()

   

• Alternatively, the above logs may show an error similar to the below:
  

  RuntimeError: CUDA out of memory occurred with warming up sample with <number> dummy requests.

Environment

Private AI Services v2.1

VMware Cloud Foundation 9.x

Cause

A known issue caused by increased VRAM requirements in the LLM engine version used in Private AI services 2.1 that may cause failures with model endpoint deployment.

Resolution

A new model endpoint will need to be deployed with an override for gpu memory utilization. The fields of an existing model endpoint are immutable and cannot be changed.

 

Option A - Create a new Model Endpoint with overrides in the VCF Automation UI

Related documentation: Deploy Completion or Embedding Model Endpoints by Using the VCF Automation UI

1. In a web browser, navigate to VCF Automation -> Build & Deploy -> Private AI -> Model Runtime and click to create a New Model Endpoint
   
   
2. Under the step Endpoint Specification, ensure that "Advanced Customization" is toggled on.
   
   
   
3. Add an argument for Flags or Arguments for Engine Customization and populate the entry with the following:
   

   --gpu_memory_utilization=0.8

   Modify the memory utilization value as needed for your specific model, but it will need to be a lower value than the amount specified in the error from Issue/Introduction.
   
   

4. Ensure that the selected vmClass and vGPU profile is sufficient for your specific model's needs.
   
   
   
5. Complete the wizard and click Deploy model endpoint.
   
   
   
6. Monitor the new model endpoint's deployment to completion.
   
   
   
7. Delete the previous, failing to deploy model endpoint from the VCF Automation web UI.

 

Option B - Use kubectl commands to create a new model endpoint with overrides

Related documentation: Deploy Completion or Embedding Model Endpoints by Using the kubectl Command

1. Connect into the PAIS cluster context where you will be deploying the new model endpoint
   
   
   
2. Create a modelEndpoint YAML with the desired vmClass and vGPU profile sufficient for your specific model's needs.
   
   
   
3. In the modelEndpoint YAML, add a "cliArgs" section under inferenceServerCustomization with the below key-value override argument:
   

   inferenceServerCustomization:
         cliArgs:
         - "--gpu-memory-utilization=0.8"

   Modify the memory utilization value as needed for your specific model, but it will need to be a lower value than the amount specified in the error from Issue/Introduction.
   
   

4. Apply the modelEndpoint YAML to deploy the modelEndpoint:
   

   kubectl apply -f <modelendpoint yaml>

   
   
   
   
5. Monitor the new modelEndpoint's deployment to completion.
   
   
   
6. Delete the previous, failing to deploy model endpoint.

Additional Information

Private AI Services 2.1 Release Notes