AI Root Cause Analysis for Prometheus Alertmanager Alerts

Alertmanager is very good at one job: it groups, deduplicates and routes alerts. It does not tell you why KubePodCrashLooping fired at 03:12. The on-call engineer still opens Grafana, runs kubectl describe, reads logs, and checks what changed. That first 10 to 20 minutes is the same work every time, and it is the part an AI agent can do before a human picks up the page.

This guide shows how to connect Alertmanager to Akmatori, an open-source, self-hosted AIOps agent, so every firing alert starts an investigation automatically. It also covers the label and grouping choices that decide whether the agent finds the cause or guesses.

How It Works

1. Alertmanager sends its normal webhook payload to an Akmatori alert source URL.
2. Akmatori normalizes each alert: labels.alertname becomes the alert name, labels.severity the severity, labels.instance the target host, labels.job the service, and annotations.runbook_url the runbook link.
3. Related alerts join one incident instead of opening ten. Akmatori first matches the same alert on the same host, then asks the model whether a new alert belongs to an open incident.
4. An agent session starts. It searches your runbooks and past incidents first, then queries your systems through the tools you connected: VictoriaMetrics, Grafana (Prometheus and Loki data sources), Kubernetes, SSH and others.
5. The result goes to the incident page and to Slack: what fired, what the agent checked, the likely root cause, and a proposed fix. Risky actions wait for a human to approve them.
6. When Alertmanager sends the resolved notification, Akmatori uses the Alertmanager fingerprint to mark the same alert resolved.

Step 1: Create the Alert Source

In Akmatori, open Connectors, choose Prometheus Alertmanager, open the Alerts tab and create a new source. Copy the webhook URL. It has the form https://<your-akmatori>/webhook/alert/<uuid>. Set a webhook secret; the full field reference is on the Alertmanager integration page.

Step 2: Add the Receiver

Add Akmatori as a webhook receiver. Akmatori accepts the secret as a bearer token:

receivers:
  - name: akmatori
    webhook_configs:
      - url: https://akmatori.example.com/webhook/alert/<uuid>
        send_resolved: true
        http_config:
          authorization:
            type: Bearer
            credentials_file: /etc/alertmanager/akmatori-secret

Keep send_resolved: true. Without it, incidents never see the recovery, and the next occurrence of the same alert has less context.

Step 3: Route Only What Deserves an Investigation

Do not send everything. An investigation costs model tokens and a few minutes of tool calls, so route the alerts a human would investigate anyway:

route:
  receiver: default
  routes:
    - receiver: akmatori
      matchers:
        - severity=~"critical|warning"
      group_by: [alertname, namespace, instance]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      continue: true

continue: true keeps your existing pager route working. Akmatori runs next to your paging, not instead of it, until you trust it.

Step 4: Test With a Fake Alert

You can send a payload by hand before you wait for a real outage:

curl -X POST https://akmatori.example.com/webhook/alert/<uuid> \
  -H "Authorization: Bearer $AKMATORI_SECRET" \
  -H "Content-Type: application/json" \
  -d '{"status":"firing","alerts":[{"status":"firing",
       "labels":{"alertname":"HighCPUUsage","severity":"critical","instance":"web-01:9100","job":"node"},
       "annotations":{"summary":"CPU above 90% for 10 minutes"},
       "fingerprint":"test-0001"}]}'

An incident appears in the dashboard and the agent starts working on web-01.

The Labels Decide the Quality

The agent reads your labels and annotations as its first evidence. Three things matter more than the model you pick:

• instance is often host:port. The default mapping uses labels.instance as the target host, so the agent sees web-01:9100. If your SSH inventory uses plain host names, add a hostname label in relabeling and change the alert source field mapping to use it. Every source has editable field mappings.
• Put the runbook in annotations.runbook_url. Akmatori stores it with the alert, and the agent reads your runbooks before it touches infrastructure. One good runbook link turns a generic investigation into your team's procedure.
• Write a real description. "CPU above 90%" says less than "CPU above 90% on the ingest workers; usual causes: backfill job, stuck consumer". The agent uses that text as its starting point.

What the Agent Can Check

With the matching connectors enabled, an Alertmanager investigation typically uses:

SSH is read-only unless you allow write commands for a host. Actions that change state become proposals that a human approves in the UI or in Slack.

Which Model to Use

Root cause accuracy depends on the model more than most teams expect. Our SRE Bench runs the same 12 real incidents on a fresh Akmatori install for each model. In the August 2026 edition, 11 models found the root cause in every scenario, and several cost under $0.10 per solved incident. If your data must stay inside your network, see self-hosted AI SRE with local models.

Conclusion

Alertmanager already knows when something is wrong. Connecting it to an agent adds the first pass of "why" before the page reaches a human: metrics, logs, pod state and history in one summary. Start with one route for critical alerts, keep send_resolved on, fix the instance label, and add runbook links. For a simpler Slack-only setup without an agent, see integrating Alertmanager with Slack.

If your team wants AI-assisted incident workflows on your own infrastructure, Akmatori helps SRE teams investigate alerts, coordinate response, and automate safe infrastructure actions. Powered by Gcore for global infrastructure reliability.