AI Services Models constraints on CPU exclusively

Products

• VMware Tanzu Platform Core

Issue/Introduction

AI Services models can show degraded response or be unable function when used exclusively on CPUs.

When trying to use the models on this conditions, the user can find errors similar the the following:

status_code: 500, model_name: gpt-oss:20b, body: Internal Server Error

Reviewing cf logs <agent_name> can show errors similar to the following

2026-05-19T13:50:34.86-0400 [APP/REV/4/PROC/WEB/0] OUT INFO:     10.###.###.###:51246 - "POST /api/chat HTTP/1.1" 200 OK
   2026-05-19T13:50:37.09-0400 [APP/REV/4/PROC/WEB/0] OUT 2026-05-19 17:50:37,090 [ERROR] __main__: SSE stream error: data: {"type":"error","errorText":"status_code: 500, model_name: gpt-oss:20b, body: Internal Server Error"}

 

Cause

A setup consistent of model running exclusively on hardware (CPU) will be constrained when handling complex prompts.

Resolution

In order to have optimal response form models on AI Services, ensure to have adequate resource allocation for models including sufficient VM types and availability of GPUs.