Nightly Dream Jobs: Agent Retrospectives That Re-Run

Every agent session leaves a trail: the commands it ran, the runbook it followed, the step it skipped, the line where it said "tests pass". Nobody reads that trail unless something breaks. A good human postmortem (see our postmortem guide) finds the gaps in process after a bad day. Agent sessions have the same gaps every day, and most of them never become an incident.

A "dream" job is a cheap way to read the trail. Once a night, a scheduled agent goes through the day's sessions and writes down what looks wrong: a runbook step that no session could follow, a test that was reported but maybe never executed, a fact that turned out stale. It does not fix anything. A second job, later, takes the notes, re-runs the evidence, and only then proposes a change.

The idea has a research cousin. The sleep-time compute paper from Letta and UC Berkeley lets models "think" offline about a context before queries arrive. On their stateful benchmarks it cut the test-time compute needed for the same accuracy by about 5x, and the authors found it works better when the next query is predictable from the context. Operations work is very predictable: the same alerts, the same runbooks, the same CI pipelines come back every week. That is why offline review pays off here.

Two Jobs, Not One

The design that holds up splits the retrospective into a reader and an actor.

The split matters because a transcript is a report, not a measurement. When the dream job reads "all tests pass", it knows that a session said so, not what the command covered. If the same job also edits the runbook, a summary of a summary turns into a rule. With two jobs, the reader can be generous with suspicion, and the actor earns every change with fresh output.

Reported vs Re-Run Evidence

Give every note an evidence class and let the class decide what the actor may do with it.

Most dream findings start as "reported". The job's value is in noticing the distance between what a session claimed and what the transcript shows it did.

Worked Example: The Fixture That Never Ran

Here is the kind of gap a dream job finds in tests. A team keeps Prometheus alert rules in a repo and tests them with promtool rule unit tests. The CI step was written a year ago, when there was one test file:

promtool test rules tests/disk_alerts_test.yml

An agent session changes the disk alert and, to cover the new behaviour, writes three new test groups: the disk fills slowly, the disk fills fast, and a scrape gap in the middle. It puts them in a new file, tests/disk_alerts_v2_test.yml, runs the CI step, and reports success. Every word of that report is true. The old fixture passed. The three new scenarios never ran, because promtool runs the test files you name on the command line, and the CI step names one.

The dream job reads the session and writes a note like this:

claim:     "promtool rule tests pass for disk alerts"   (reported)
command:   promtool test rules tests/disk_alerts_test.yml   (observed)
gap:       session created tests/disk_alerts_v2_test.yml with 3 test groups;
           no command in the session or CI names that file
next step: re-run with both files; prove each new group executes

The follow-up job then re-runs it. The Prometheus docs give it the tools it needs: the unit testing guide shows promtool test rules taking several files in one call, and the promtool reference documents --run, which runs only the test groups whose names match a regular expression.

# Run everything, the way CI should
promtool test rules tests/disk_alerts_test.yml tests/disk_alerts_v2_test.yml

# Prove one new scenario runs on its own
promtool test rules --run 'disk_fills_fast' tests/disk_alerts_v2_test.yml

One more check closes the loop: break an expected label in a new group on purpose and confirm the run fails. A test that cannot fail is not evidence. This catches a second trap from the same docs. To assert that an alert does not fire, you leave exp_alerts empty. If the rule itself is broken and never fires, those negative tests keep passing.

The actor's output is a change to the CI step (a glob or an explicit list, and a check that fails when a *_test.yml file is not listed), with the re-run log attached. That is a reviewable proposal, not a hunch.

What Dream Jobs Find in Runbooks

Tests are the sharpest case, but runbooks give a steadier stream of findings. Point the prompt at these patterns:

• Steps nobody can follow. A runbook says to check a dashboard or a command, and every session that opened the runbook did something else. The step is stale or the tool moved.
• Searches that found nothing. A session searched the runbook library for an alert name and got no hit. That is a missing runbook, and the transcript already holds a draft of it.
• Repeated discovery. Three sessions this week found the same fact the slow way (which host holds the replica, which flag disables the cache). That fact belongs in memory or in the runbook.
• Contradicted facts. A memory says one thing, and today's session saw another on a live system. The memory needs a re-check, not a silent overwrite.

Running It on Akmatori

Akmatori already has the memory half of this loop. As the architecture docs describe, alert investigations do not write memory. When an incident closes, a post-incident memory pass reads the runs, the alerts and the final answers, and saves at most five memories. Each memory carries an evidence grade: confirmed (only an operator can set it), observed (seen directly), or unverified. An agent's causal claim is always unverified, which is the "reported" class from the table above. Akmatori records which memories each run searched, read and cited, and the Dreaming cron job can retire unverified agent memories that nobody used for 90 days.

For the runbook and test review, add your own cron jobs. A cron job has a five-field schedule, a prompt, tools_enabled (all enabled connector tools, or none) and builtin_tools (any of events, incidents, proposals, runbooks, memory). The reader gets no connector tools:

curl -X POST https://your-akmatori-instance/api/cron-jobs \
  -H "X-API-Key: $AKMATORI_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "name": "Nightly retrospective",
    "schedule": "0 3 * * *",
    "prompt": "Read incidents closed in the last 24 hours. For each claim of a passing test, a rollback or a verification, record the claim, the exact command if the transcript shows one, and the gap. Save notes as memories. Do not change runbooks.",
    "tools_enabled": false,
    "builtin_tools": ["incidents", "runbooks", "memory"],
    "enabled": true
  }'

The actor runs at 0 7 * * * with tools_enabled: true so it can reach your repos and CI through the connectors you have set up (an SSH or HTTP connector, for example), and with memory and runbooks to read the notes. Proposal review fits the last row of the evidence table: the API lets an operator read a proposal with the evaluator's rationale, question it in a review chat, and approve or reject it. The self-improvement evaluator runs without connector tools, with only the incidents and proposals tools, so it drafts changes from evidence without touching infrastructure.

For more on what an incident agent should remember and forget, see incident memory for AI SRE agents.

Checklist Before You Trust a Nightly Finding

• The note names the claim, the command (if any), and the gap as separate fields.
• The actor re-ran the command today, and the output is attached.
• The re-run covers what the change touched: every new file, every new test group.
• At least one deliberate failure proved the test can fail.
• The reader had no write access to runbooks, CI or hosts.
• A human approved anything that changes shared memory or a runbook.
• Notes nobody acted on expire, so last month's suspicion does not become this month's fact.

Conclusion

A nightly retrospective is cheap: one read-only scheduled agent reading what already happened. Its findings are only as good as the re-run behind them. Keep the reader and the actor apart, grade every note as reported until a fresh command says otherwise, and "tests pass" becomes a question your pipeline can answer.

If your team wants AI-assisted incident workflows with memory that keeps its evidence straight, Akmatori helps SRE teams investigate alerts, coordinate response, and automate safe infrastructure actions. Powered by Gcore for global infrastructure reliability.