AI Root Cause Analysis for PagerDuty Incidents

PagerDuty makes sure the right person wakes up. It does not tell that person what broke. The first minutes after the page are spent finding the alert source, opening dashboards and checking recent deploys, often on a phone. If an agent has already done that work when the engineer opens the incident, the response starts from a hypothesis instead of from zero.

This guide connects PagerDuty to Akmatori, an open-source, self-hosted AIOps agent. PagerDuty keeps doing the paging. Akmatori receives the same incidents and investigates them in parallel.

How It Works

1. A PagerDuty v3 webhook subscription sends incident.triggered events to Akmatori.
2. Akmatori normalizes the incident: the title becomes the alert name, urgency and priority become the severity, and the service name and ID become labels. A runbook link in the event details is stored with the alert.
3. The agent searches your runbooks and past incidents, then investigates with the tools you connected: PagerDuty itself, your metrics, Kubernetes and SSH.
4. The findings go to the incident page and Slack. With the PagerDuty tool, the agent can also add them as a note on the PagerDuty incident.
5. incident.resolved marks the alert resolved in Akmatori.

Step 1: Create the Alert Source

In Akmatori, open Connectors, choose PagerDuty, open the Alerts tab and create a source. Copy the webhook URL. The PagerDuty integration page shows an example payload.

Step 2: Add a Webhook Subscription

In PagerDuty, go to Integrations > Generic Webhooks (v3) and add a subscription:

• Scope: start with one service, not the whole account.
• Webhook URL: the Akmatori URL.
• Events: incident.triggered and incident.resolved.
• Custom header: Authorization: Bearer <your-secret> if you set a secret in Akmatori.

Do not subscribe to incident.acknowledged. Akmatori v0.4.0 and earlier treat an acknowledged incident as resolved, so the alert would close as soon as someone acknowledges the page, before the problem is fixed. Later versions ignore it. If PagerDuty can reopen incidents in your setup, also subscribe to incident.reopened.

Step 3: Connect PagerDuty as a Tool

The webhook tells Akmatori that an incident exists. To let the agent read and update PagerDuty, enable the PagerDuty tool connector with a REST API token. The useful tools during an investigation are:

Question	Tool
What alerts are behind this incident?	pagerduty.get_incident_alerts
What changed recently on this service?	pagerduty.list_recent_changes
Who is on call, and what is the escalation path?	pagerduty.get_on_calls, pagerduty.get_escalation_policies
Did the responders already write notes?	pagerduty.get_incident_notes
Put the findings where the on-call works	pagerduty.add_incident_note

The tool can also acknowledge, reassign and resolve incidents. If you want the agent to only read and write notes, use a token for a user with limited rights, and say in your skill instructions that the agent must not change the incident state.

The Trap: An Incident Is Not an Alert

A PagerDuty incident payload describes the service and the incident, not the failing host or pod. "Checkout API: high error rate" on service checkout does not say which cluster, namespace or node. Two things give the agent the missing context:

• Alert details. The monitoring alert that created the incident usually has the host, the query and the labels. The agent reads them with pagerduty.get_incident_alerts, so keep those fields in your monitoring payloads instead of stripping them.
• Runbooks per service. Write one runbook for each PagerDuty service: where it runs, which dashboards matter, what usually breaks. Akmatori searches runbooks before it touches infrastructure, and a service-to-cluster mapping in a runbook saves the agent several guesses.

If your alerts come from Prometheus or Grafana before they reach PagerDuty, you can also send them to Akmatori directly. The direct payload has every label. See the guides for Alertmanager and Grafana.

What the On-Call Sees

A good investigation note answers four questions in a few lines: what fired, what the agent checked, the most likely cause with evidence, and the suggested next step. Akmatori proposes risky actions, such as a restart or a rollback, as proposals that a human approves in the UI or in Slack. Nothing changes in production because a model decided it should.

Choosing a Model

The quality of that note depends on the model. SRE Bench runs 12 real incidents on a fresh Akmatori install per model. In the August 2026 edition, 11 models found the root cause in all 12, several for less than $0.10 per solved incident. For models you host yourself, read self-hosted AI SRE with local models.

Conclusion

PagerDuty gets a human to the incident. Akmatori gets the first diagnosis there at the same time. Subscribe to triggered and resolved events only, give the agent read access to incident alerts and recent changes, write a runbook per service, and let it post its findings as an incident note.

If your team wants AI-assisted incident workflows on your own infrastructure, Akmatori helps SRE teams investigate alerts, coordinate response, and automate safe infrastructure actions. Powered by Gcore for global infrastructure reliability.