Architecture

How one incident moves through the system

1. An alert arrives. Alertmanager, Zabbix, PagerDuty, Grafana or Datadog posts to /webhook/alert/{instance_uuid}. The proxy passes it to the API server, which checks the per-instance secret and normalizes the alert.
2. The API server opens an incident. It either attaches the alert to an open incident or creates a new one, writes AGENTS.md for the run, and posts the alert to Slack, Telegram or Discord.
3. The agent worker gets the task. Over a WebSocket, the API server sends the incident, the skills, the LLM settings and the run's tool allowlist.
4. The agent investigates. It runs the LLM loop against your provider. The worker is the only container that calls the LLM.
5. Tools run in the gateway. Each gateway_call goes to the MCP Gateway, which checks the allowlist, loads the connector's credentials from PostgreSQL, and calls your system over SSH, SQL, HTTPS or MCP.
6. The answer goes back. Output streams to the API server, which saves the run and posts the root cause in the incident's chat thread.
7. Memory is written after close. When an alert incident closes, a post-incident memory pass reads the runs and saves at most five graded, redacted memories for the next investigation.

Containers

The Docker Compose stack starts these services. Only the proxy publishes a port: ${HTTP_PORT:-8080}.

Instructions and tools

Agents read their instructions from files that Akmatori generates for each run. Tools are not assigned to skills: every enabled connector is available to an incident run, and the generated connected-tools skill lists them with their parameters and gateway_call(...) examples.

## Production PostgreSQL (logical_name: "prod-db", type: postgresql)

Usage (via gateway_call):

```python
gateway_call("postgresql.list_tables", {}, "prod-db")
gateway_call("postgresql.describe_table", {"table_name": "users"}, "prod-db")
```

For multi-step work, use `execute_script` with built-in `gateway_call()`.

Tool authorization

A skill does not restrict tools. The API server builds each run's allowlist from the kind of run and sends it with the task, and the gateway refuses every tool outside it. To keep a connector away from agents, disable its tool instance.

Remote MCP servers

• Streamable HTTP only, with bearer, API-key header or OAuth 2.1 auth. OAuth tokens are stored in PostgreSQL.
• You can switch individual tools off. Tool schemas are refreshed every five minutes.
• Outbound connections refuse loopback, link-local and cloud-metadata addresses, and redirects are not followed. Set MCP_BLOCK_PRIVATE_NETWORKS to refuse private ranges too.
• A reload reconnects only the servers whose URL, prefix, auth or disabled tools changed.

Investigation guardrails

The incident manager prompt makes context recall a fixed first step. For every incident, the agent searches runbooks, then cross-incident memory, then recent change events, and only then calls infrastructure tools.

1. Understand the problem.
2. Search runbooks first:
   gateway_call("runbooks.search", {"query": "<the original alert text>"})
   then gateway_call("runbooks.get", {"id": <id>}) for the best match.
3. Search cross-incident memory next:
   gateway_call("memory.search", {"query": "<the same query>", "scopes": ["<skill names relevant to this alert>", "global"]})
   then gateway_call("memory.get", {"id": <id>}) for a relevant memory.
4. Check recent changes:
   gateway_call("events.search", {"kind": "change", "since": "<60 min before the alert>", "host": "<host>"}, "events")
5. Read the relevant SKILL.md files and connected-tools/SKILL.md, then call infrastructure tools.

Runbooks and memory

Runbooks and memories live in PostgreSQL. Operators write runbooks and can add, edit or delete any memory in the UI. Agents use them through the runbooks.* and memory.* gateway tools.

• Alert investigations do not write memory. When an alert incident closes, the post-incident memory pass reads the runs, the alerts and the final answers, and saves at most five memories. A failed pass is retried up to three times with backoff, and you can run it again from the incident page.
• Other work can write memory directly. Request-driven runs save memories with memory.write.
• Evidence grades. Each memory is confirmed (only an operator can set it), observed (seen directly) or unverified. An agent's causal claim is always unverified.
• Protected rows. An agent cannot change operator memories or context-file summaries.
• Usage. Akmatori records which memories each run searched, read and cited. The Dreaming cron job can retire unverified agent memories that nobody used for 90 days.
• Secrets. Private keys, tokens, URL passwords and similar values are redacted on a best-effort basis before a memory is saved.
• From chat. In Slack, reply remember: … in a thread to save an operator memory. You can turn memory generation off in Settings → Memory; operator writes still work.

Scheduled agents

Cron jobs use the same execution path as incidents. A cron job posts to its configured channel or falls back to the workspace default. Two fields control its tools: tools_enabled (all enabled connector tools, or none) and builtin_tools (which of events, incidents, proposals, runbooks and memory it can call). New cron jobs default to connector tools on, with events, runbooks and memory.

The self-improvement evaluator is a system cron job, off by default. It runs without connector tools and with the events, runbooks, memory, incidents and proposals tools, so it can review recent evidence and draft runbook and skill updates as proposals.

Design principles

How the tool docs are generated

1. You add, enable or disable a connector in the UI.
2. The backend regenerates connected-tools/SKILL.md from every enabled tool instance, with its logical name and usage examples for its tool type. It also re-renders the file on a timer.
3. Every other SKILL.md keeps its own instructions and ends with its memory manifest and a pointer to connected-tools.
4. HTTP connector and MCP server changes trigger a gateway reload, so new tools appear without a rebuild.
5. When an incident starts, Akmatori writes AGENTS.md with the incident workflow, the global memory manifest and the list of context files.

Contributor checklist

Troubleshooting