Open WebUI agents fail to execute correctly when using Model Context Protocol (MCP) through Private AI Services

Products

• VCF Private AI Services

Issue/Introduction

1.Parsing errors occur when the inference server attempts to process tool calls on large model contexts:

"error": "Parsing inference server response failed"

2.Agent work fine if tool calling disabled

3.Model endpoint name is gemma-4-31b-it

4.vLLM version is already 0.19.1

Environment

VCF Private AI Service 2.x

Resolution

Modify the yaml of gemma model endpoint to support the tool calling correctly, below lines should be inserted to model endpoint yaml refer to below example:

 inferenceServerCustomization:

      cliArgs:

        ...

        - "--enable-auto-tool-choice"

        - "--tool-call-parser=gemma4"

        ...