AI Root Cause Analysis for Grafana Alerting

Grafana Alerting tells you that a query crossed a threshold. The investigation that follows is almost always the same: open the dashboard, widen the time range, check the logs panel, compare with the deploy annotations, and then go to the host or the cluster. Most of that work happens inside Grafana itself, which makes it a good fit for an AI agent that can read the same data sources.

This guide connects Grafana Alerting to Akmatori, an open-source, self-hosted AIOps agent. Every alert that reaches the Akmatori contact point starts an investigation, and the agent uses Grafana's own APIs to look at metrics, logs and dashboards.

How It Works

1. A Grafana notification policy sends the alert to a webhook contact point that points at Akmatori.
2. Akmatori normalizes the payload: labels.alertname is the alert name, labels.severity the severity, labels.instance the target host, and annotations.runbook_url the runbook link.
3. Alerts for the same problem join one incident instead of opening one each.
4. The agent searches your runbooks and past incidents, then queries Grafana and your other connected systems.
5. The incident page and Slack get the summary: evidence, likely root cause and a proposed fix. Risky actions need human approval.

Step 1: Create the Alert Source

In Akmatori, open Connectors, choose Grafana, open the Alerts tab and create a new source. Copy the webhook URL and set a webhook secret. The Grafana integration page lists the payload fields.

Step 2: Add a Webhook Contact Point

In Grafana, go to Alerting > Contact points > Add contact point and choose Webhook. Paste the Akmatori URL. For the secret, use bearer authorization, or add an X-Grafana-Secret header with the same value. Akmatori accepts both.

If you manage Grafana as code, the same contact point in a provisioning file looks like this:

apiVersion: 1
contactPoints:
  - orgId: 1
    name: akmatori
    receivers:
      - uid: akmatori-webhook
        type: webhook
        settings:
          url: https://akmatori.example.com/webhook/alert/<uuid>
          httpMethod: POST
          authorization_scheme: Bearer
          authorization_credentials: $AKMATORI_SECRET

Step 3: Add a Notification Policy

Create a child policy that matches the alerts you want investigated, for example severity =~ critical|warning, and send them to the akmatori contact point. Turn on Continue matching if the same alerts must also reach your existing on-call contact point.

Step 4: Connect Grafana as a Tool

The alert source receives alerts. To let the agent read your data, also enable the Grafana tool connector with a service account token. With it, the agent can:

The Grafana tool can also write: it can create an annotation and create a silence. Use a service account with the smallest role that covers what you want the agent to do. If the agent should only read, give it the Viewer role.

The Trap: Aggregated Queries Lose Their Labels

Grafana alert rules often use queries like sum(rate(http_requests_total{status=~"5.."}[5m])). The sum() removes every label, so the alert reaches Akmatori with only alertname, and no instance, namespace or service. The agent then has to guess where to look.

Keep at least one locating label in the query:

sum by (namespace, service) (rate(http_requests_total{status=~"5.."}[5m]))

Or add static labels to the rule, such as service=checkout and team=payments. Add a runbook_url annotation too. Akmatori stores it with the alert, and the agent reads your runbooks before it touches infrastructure. These two changes improve the result more than switching to a larger model.

Beyond Grafana

Grafana shows what the metric did. It does not always show why. With more connectors enabled, the same investigation continues outside Grafana: kubernetes.get_pod_logs and kubernetes.get_events for the cluster, read-only ssh.execute_command for a host, and VictoriaMetrics for long-range metric queries. The agent decides which tools to call from the alert and the runbook.

Choosing a Model

The model choice changes how often the agent finds the real cause. SRE Bench measures this on 12 real incidents, with a fresh Akmatori install for each model. In the August 2026 edition, 11 models solved all 12, and several of them did it for less than $0.10 per solved incident. If you already run Grafana on your own servers and want the model there too, read self-hosted AI SRE with local models.

Conclusion

Grafana already holds the data an investigation needs. Sending its alerts to an agent with read access to that data turns "threshold crossed" into a first diagnosis before a human opens the dashboard. Keep locating labels in your alert queries, add runbook links, and give the service account only the access you need. For the Prometheus side of the same setup, see AI root cause analysis for Alertmanager alerts.

If your team wants AI-assisted incident workflows on your own infrastructure, Akmatori helps SRE teams investigate alerts, coordinate response, and automate safe infrastructure actions. Powered by Gcore for global infrastructure reliability.