Local Models: Ollama, vLLM and LiteLLM

How the Custom provider works

The agent worker sends POST <Base URL>/chat/completions with streaming and function tools. The investigation agent needs tool calls: it reaches SSH, Kubernetes, Zabbix, metrics and every other connector through one tool, gateway_call. A server or a model without working tool calls can write a summary, but it cannot investigate.

Add the model in Settings → AI models → Add Configuration:

Save the model and click Activate. Investigations, chat and the short one-shot calls (titles, summaries, alert extraction) then all use it.

Ollama

The simplest setup runs Ollama as one more container on the network that the agent worker uses for outbound calls. Put this file next to docker-compose.yml:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: akmatori-ollama
    restart: unless-stopped
    environment:
      # The default is 4k tokens without a large GPU: far too small.
      - OLLAMA_CONTEXT_LENGTH=65536
    volumes:
      - ollama_models:/root/.ollama
    networks:
      - frontend
    # GPU (NVIDIA Container Toolkit):
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]

volumes:
  ollama_models:

Start it and pull a model:

docker compose up -d ollama
docker exec akmatori-ollama ollama pull qwen3:30b-a3b
docker exec akmatori-ollama ollama list

Then add a Custom model with Base URL http://ollama:11434/v1, API key ollama and the model name that ollama list prints.

Ollama on the host instead

If Ollama already runs on the Docker host, three things must be true:

1. Ollama listens on an address that containers can reach. By default it listens on 127.0.0.1:11434 only; set OLLAMA_HOST=0.0.0.0:11434 (or the Docker bridge address, for example 172.17.0.1:11434) and restart it. Set OLLAMA_CONTEXT_LENGTH there too (see the warning above).
2. The worker can resolve the host. On Linux, host.docker.internal does not exist by default; add it to akmatori-agent (the only container that calls the model):

services:
  akmatori-agent:
    extra_hosts:
      - "host.docker.internal:host-gateway"

3. The host firewall accepts the connection. Traffic from a container to the host goes through the host's INPUT chain. With a default-deny policy the request hangs and Akmatori reports Request timed out. Allow the Docker subnets to port 11434, for example iptables -I INPUT -s 172.16.0.0/12 -p tcp --dport 11434 -j ACCEPT.

The Base URL is then http://host.docker.internal:11434/v1.

vLLM and SGLang

vLLM serves the OpenAI API at /v1. It returns tool calls only when you start it with automatic tool choice and the tool-call parser for your model family:

vllm serve Qwen/Qwen3-32B \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --api-key change-me

See the vLLM tool calling guide for the parser of each model family. Set the API key to the value of --api-key, or to any value if you did not set one. Akmatori sends the system prompt with the role system, not developer, to custom endpoints, because some chat templates drop the developer role without an error.

LiteLLM proxy

A LiteLLM proxy gives Akmatori one OpenAI-compatible endpoint in front of many backends (Ollama, vLLM, Azure OpenAI, Bedrock, Vertex AI) with central keys, budgets and logs. Define the models in the proxy config:

model_list:
  - model_name: sre-agent
    litellm_params:
      model: ollama_chat/qwen3:30b-a3b
      api_base: http://ollama:11434

litellm_settings:
  # Required: drops OpenAI-only fields (store) that ollama_chat rejects.
  drop_params: true

In Akmatori, add a Custom model:

• Base URL: the proxy address with /v1, for example http://litellm:4000/v1
• API key: a LiteLLM virtual key, or the master key
• Model: the model_name from the config, for example sre-agent

Keep drop_params: true. Akmatori sends OpenAI request fields such as store, and without this setting LiteLLM rejects every request to an ollama_chat/ model with 400 UnsupportedParamsError: ollama_chat does not support parameters: ['store']. With it, tool calls and thinking text pass through the proxy (tested with LiteLLM 1.108.0 and Ollama 0.40.15).

The backend model behind the alias must support tool calls, and the Ollama context-length warning above applies behind the proxy too.

Choose a model that can call tools

Akmatori gives the model one generic tool, gateway_call(tool_name, args, instance), instead of one tool per command. The model must put the inner tool name (for example ssh.execute_command) into the tool_name argument. Small models often call ssh.execute_command as if it were a tool of its own.

We sent the worker's own request (the gateway_call schema, a disk-usage alert, streaming) to Ollama 0.40.15 with each Qwen3 size, directly and through LiteLLM:

Every failed request ended with an empty answer. Thinking level, context length and the LiteLLM proxy did not change the rate. So even a 30B model on a single machine is a demo, not a production setup: about half of its investigations would stop at the first tool call. The open-weight models that do the job are much larger; for the ones that found the root cause in 11 of 12 real incidents and the hardware they need, see Self-Hosted AI SRE Agent With Local Models and SRE Bench. Test your model with a few alerts before you rely on it.

Troubleshooting