Using Operations Manager VMs to host GenAI gguf files

Products

• VMware Tanzu Platform Core

Issue/Introduction

When using the GenAI on Tanzu Platform tile, you may want to host your gguf files on the Ops Manager VM for an offline deployment.

https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/ai-services/10-0/ai/tutorials-offline-model-support.html#-step-4:-preparing-ollama-models

Please keep in mind that these steps will not persist through an Ops Manager upgrade. As such, they should mainly be used for testing purposes.

Environment

AI Services

Resolution

On the Operations Manager VM, run the following steps:

sudo mkdir -p /opt/models

Place your needed .gguf files into this directory

sudo chmod 644 /opt/models/*

Once the .gguf files are located in /opt/models:

cd /etc/nginx

vim nginx.conf

At the bottom, add in the location for the gguf files. Like so:

    location = /Phi-4-mini-instruct-Q8_0.gguf {
      alias /opt/models/Phi-4-mini-instruct-Q8_0.gguf;
      default_type application/octet-stream;
    }

Do this for each gguf file you are hosting here.

Then run:

sudo nginx -t (validate the config)

sudo systemctl reload nginx

Once this has been set up, configure your AI services tile as per step 4.5 here:

https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/ai-services/10-0/ai/tutorials-offline-model-support.html#4.5-deploying-ollama-models

Finally, run Apply changes against the AI services tiles, with the errands enabled.