vllm-worker job failed to start with 'DeepGEMM backend is not available or outdated' error

Products

• VMware Tanzu Platform

Issue/Introduction

When updating AI Services tile 10.4.1 to 10.4.2 the vllm-worker job failed to start with the following errors seen in log file - /var/vcap/sys/log/vllm-worker/vllm-worker.stdout.log on vllm-worker instance.
  
  (EngineCore pid=28726) ERROR 06-19 03:01:55 [core.py:1165] RuntimeError: Worker failed with error 'DeepGEMM backend is not available or outdated. Please install or update the `deep_gemm` to a newer version to enable FP8 kernels.', please check the stack trace above for the root cause

 

Environment

Tanzu Hub 10.4.2

Cause

The error was due to a bug in VLLM which is being traced through https://github.com/vllm-project/vllm/issues/41849

 

Resolution

It's worth noting that Tanzu AI Services product is designed to deploy the version of vLLM / Ollama that is shipped with the tile to the platform, apply the configuration that the user put in the tile and attempt to start the service. If there are issues with the configuration being applied resulting in the vLLM /Ollama failing to start, it's on the users to debug the required configuration needed to run the model on the hardware they have chosen. Therefore, it's expected the "DeepGEMM backend is not available or outdated" error to be fixed by VLLM. 

Before the issue is fixed, it can be worked around by setting environment variable VLLM_USE_DEEP_GEMM=0 before starting the VLLM service. This environment variable can be set to the VLLM models added in AI Services tile as shown by image below. 

Additional Information

AI Services Support Information

Support Scope for Model Deployment

Broadcom supports the deployment and configuration of specific released versions of vLLM and Ollama on the AI Services platform, on GPU hardware supported by your chosen infrastructure provider (such as vSphere, AWS, Azure, or GCP). This includes configuration changes made through the AI Services tile in Tanzu Operations Manager.

The following are outside the scope of Broadcom support:

• Selecting a model appropriate for your workload or use case
• Validating that a specific model is compatible with your chosen hardware
• Debugging failures caused by custom model configurations, advanced engine arguments, or environment variable overrides
• Performance tuning or capacity planning for your specific use case

Each release includes a set of known valid configurations that were tested against specific models, model providers, and hardware at the time of release. These configurations are provided for reference only and do not constitute a guarantee of compatibility or performance for all deployments.

Customers are responsible for validating their chosen model, hardware, and configuration in a staging environment before deploying to production. Customers must also ensure that their infrastructure provider supports the GPU hardware they intend to use.