Self-Hosted AI SRE Agent With Local Models

An AI SRE agent sees everything an engineer sees during an incident: logs, metrics, host names, configuration, sometimes customer data in error messages. For many teams, sending that to a public model API is not allowed. Banks, healthcare, public sector, and companies with data sovereignty rules need the whole loop, agent and model, inside their own network.

Akmatori is an open-source AIOps agent built for this case. It runs on your servers with Docker Compose and talks to any OpenAI-compatible endpoint, so the model can run on your own GPUs. This guide covers which open models are good enough, what hardware they need, and how to connect them.

Are Open Models Good Enough?

Yes, if you pick the right ones. We measure this with SRE Bench: 12 real incidents, a fresh Akmatori install per model, and a pinned judge that grades each investigation against written ground truth. In the August 2026 edition, these models have published weights on Hugging Face, and each found the root cause in 11 of 12 incidents:

For comparison, the best closed models solved 12 of 12. The gap is one scenario, and the open models are close behind. Read each license before you deploy: "open weights" does not always mean unrestricted commercial use.

Two honest limits of these numbers. First, we ran these models through hosted APIs, not on our own GPUs, so a quantized local copy can score lower than the full model. Second, we have not yet benchmarked small models, such as 7B to 30B models that fit on one GPU. For narrower tasks with small models, see our post on Qwen3 for local SRE triage.

What Hardware They Need

The models that pass SRE Bench are large mixture-of-experts models. Only a fraction of the parameters is active per token, but all weights must be in memory. A simple rule: at FP8, the weights take about one byte per parameter, and at 4-bit about half a byte.

• MiniMax M3.1 Flash (about 220B): roughly 220 GB at FP8, or about 110 GB at 4-bit, plus memory for the context. This is the smallest model on the list and the most realistic first local deployment.
• DeepSeek V4.1 Flash and GLM-5.2 (about 750B): roughly 750 GB at FP8. That is a full multi-GPU server.
• Kimi K3 (about 2.8T): a multi-node deployment for most teams.

Many companies already have a shared inference platform or a sovereign cloud with these models. If yours does, point Akmatori at it instead of running a new model server.

Connect Akmatori to Your Model

Akmatori's custom provider works with any OpenAI-compatible endpoint: vLLM, SGLang, Ollama, llama.cpp server, LiteLLM, or an internal gateway. In the Akmatori settings, choose the Custom provider, enter the base URL of your endpoint (for example http://llm.internal:8000/v1), the model name it serves, and an API key if your gateway needs one.

A minimal vLLM server with tool calling looks like this:

vllm serve MiniMaxAI/MiniMax-M3.1-Flash \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser <parser-for-your-model> \
  --max-model-len 131072

Check the model card for the right tool-call parser and recommended flags.

Three Traps

Tool calling must work. An SRE agent is a loop of tool calls: query metrics, read logs, run a read-only command. If the server does not parse tool calls for your model, the agent gets text instead of calls and the investigation stops early. In vLLM, that means --enable-auto-tool-choice and the right --tool-call-parser. Test one tool call before you test a full incident.

The context window must be large. An investigation collects a lot of output: log lines, metric series, kubectl describe text. A small context window cuts the evidence. Ollama in particular uses a small default context; raise it with OLLAMA_CONTEXT_LENGTH or the model's num_ctx. Aim for at least 64k tokens.

Quantization has a cost. A 4-bit model is much cheaper to serve, but it can lose exactly the careful reasoning that separates 11/12 from 9/12. Run the same few incidents on the quantized and the full model before you decide.

Keep the Rest Inside Too

A local model is only half of the data boundary. Akmatori itself runs on your servers. The agent reaches your systems through tools you connect: read-only SSH by default, Kubernetes, Zabbix, Grafana, VictoriaMetrics and others. Risky actions become proposals that a human approves. For outbound traffic that must go through a corporate proxy, Akmatori supports HTTP_PROXY and per-tool proxy settings. Slack is optional: if chat messages must not leave the network either, use the web dashboard instead.

Conclusion

A self-hosted AI SRE agent with local models is practical today. Several open-weight models find the root cause in 11 of 12 SRE Bench incidents, Akmatori connects to them through any OpenAI-compatible endpoint, and the main traps are tool calling, context length and quantization. Start with one alert source, such as Alertmanager, and compare the local model with a hosted one on your own incidents. If you are comparing self-hosted with SaaS options, see Akmatori vs Resolve.ai.

If your team wants AI-assisted incident workflows on your own infrastructure, Akmatori helps SRE teams investigate alerts, coordinate response, and automate safe infrastructure actions. Powered by Gcore for global infrastructure reliability.