Building a Durable AIOps On-Call Agent with Eve.dev

It's 3:07 AM. A latency alert fires on the payments service. The primary on-call engineer is asleep, acknowledges four minutes later, spends eight minutes finding the runbook, and another ten confirming it's a connection-pool exhaustion issue they've seen twice before. Twenty-two minutes in, they finally run the fix everyone already knew they'd run.

Most of that time wasn't problem-solving. It was detection lag, acknowledgement lag, and re-deriving context a machine could have assembled instantly. That gap is exactly where an on-call agent earns its keep—not by replacing the engineer's judgment on the scary decisions, but by doing the tedious first-responder work and handing over a clean, pre-diagnosed incident.

The catch is that incidents are long, messy, and stateful. An agent that forgets everything when your container gets rescheduled is worse than no agent at all. So this build is really about two things at once: a competent first responder, and a workflow durable enough to be trusted with a real outage.

What an AIOps on-call agent actually does

An AIOps on-call agent is an autonomous responder that sits at the front of your incident process: it receives alerts, decides how serious they are, gathers evidence, and drives the response either to resolution or to the right human. It's not a chatbot bolted onto PagerDuty. It's a workflow that walks the same path a good engineer would.

That path has a well-worn shape. PagerDuty models the incident response lifecycle as five stages, and our agent maps onto them almost one-to-one:

1. Detect — an alert arrives from monitoring, an internal team, or a support escalation.
2. Triage — classify severity and customer impact, urgency, and which services and stakeholders are affected.
3. Diagnose — pull logs, metrics, and recent deploys to form a hypothesis.
4. Remediate — apply a fix (or propose one and wait for sign-off).
5. Continuous learning — capture what happened so the next incident is faster.

One distinction matters before you write a line of code: an alert is not an incident. An alert is a signal—a threshold crossed, an anomaly detected. Triage is what decides whether that signal deserves to become an incident. Treating every alert as an incident is how teams burn out, and it's a mistake an agent will happily make at scale if you let it. The triage step exists precisely to filter.

Get triage wrong and everything downstream breaks. Misclassify a genuine P1 as a P3 and the escalation chain never fires: the comms lead is never looped in, stakeholders never hear about it, and the engineers who should be awake stay asleep. So triage isn't a nicety in this design. It's the load-bearing decision.

Why durability is non-negotiable here

Here's the scenario that kills naive agents. Your agent is 25 minutes into a database incident. It has already ruled out three causes, confirmed a fourth, and is waiting for an engineer to approve a failover. Then your platform reschedules the pod the agent runs in—a routine deploy, a node drain, a spot-instance reclaim. A stateless agent wakes up with amnesia. It either does nothing or, worse, starts over and re-runs diagnostics that mutate state.

Eve.dev handles this by treating the agent's execution as a checkpointed state machine. Every tool call and every state transition is persisted. When the agent needs to wait—for a diagnostic to finish, for a human to approve, for an escalation timer to elapse—the workflow is suspended to durable storage. When the trigger arrives, Eve.dev rehydrates the agent with its full memory intact. We used the same suspend() primitive to build a crash-resistant support agent; incident response is where that durability stops being a nice-to-have and becomes the whole point.

The practical test: could the machine your agent runs on be rebooted at any second during an incident, and would the incident still be handled correctly? With Eve.dev, yes. That's the bar.

The one rule that keeps this safe: blast radius × confidence

Before building anything, we need a governance rule, because "an AI that restarts production services on its own" is a sentence that should make you nervous. It should make the agent nervous too.

The model that holds up across the incident-response literature is simple and worth internalizing: decide what to automate by blast radius and detection confidence, not by confidence alone.

• Low blast radius + high confidence → let the agent act unattended. Clearing a filled-up temp directory, restarting a single stateless worker, flushing a cache.
• High blast radius → keep it manual at any confidence level. Production databases, VIP-customer-facing services, revenue-critical paths, credential rotation at scale. Even if the model is 99% sure, the cost of the 1% is unbounded.
• Low confidence → don't automate at all yet. Tune the signal first.

The reason high blast radius stays human regardless of confidence is the failure mode that should keep you up at night. It isn't the agent crashing. It's the agent successfully executing exactly the right remediation on exactly the wrong diagnosis—a clean, logged, technically flawless action that happened to be aimed at the wrong target. When that happens with a service restart, you lose a few seconds. When it happens with a database failover or a mass credential rotation, you've turned a small incident into a large one, and there's no defensible answer to "why did we let it do that automatically?"

So the rule we'll encode: automated actions must be safe, reversible, and logged. Anything that isn't all three goes through a human approval gate. Everything in this build follows from that one sentence.

Project setup

Start with a fresh project and the Eve core package.

npx create-next-app@latest oncall-agent
cd oncall-agent
npm install @eve/core

Eve.dev uses a filesystem-first layout—your agent's brain lives in folders, the same way routes live in folders in Next.js. If you've read the filesystem-based workflow guide, this will feel familiar. Create the directory for our on-call agent:

mkdir -p eve/agents/oncall/tools

We'll end up with this structure:

eve/
└── agents/
    └── oncall/
        ├── instructions.md      # how the agent triages and when it may act
        ├── schedule.ts          # proactive health sweep
        └── tools/
            ├── get_alert_context.ts   # read-only
            ├── run_diagnostics.ts     # read-only
            ├── request_approval.ts    # the human gate
            ├── remediate.ts           # gated actions
            └── escalate.ts            # durable escalation timer

Step 1: Write the operating instructions

Eve.dev parses Markdown into the agent's system prompt. This file is where the governance rule stops being a blog paragraph and becomes policy the agent has to follow. Create eve/agents/oncall/instructions.md:

# On-Call Response Agent

You are a first-responder for production incidents. Your job is to triage
alerts, diagnose the likely cause, and drive each incident toward resolution
—either by fixing safe problems yourself or by escalating to a human with a
clear summary.

## Triage first, always
An alert is not an incident. For every alert:
1. Classify severity P0–P3 based on customer impact and scope.
2. Decide whether it warrants an incident at all. Transient blips that
   self-recover do not.
3. Never downgrade an ambiguous signal to avoid work. If unsure between two
   severities, pick the higher one.

## What you may do without asking
You may run any READ-ONLY tool freely: pulling logs, metrics, recent deploys,
and health checks. Diagnose thoroughly before proposing any action.

## What you must get approved
You may NOT take any action that is not safe, reversible, and logged.
Specifically, ALWAYS call `request_approval` before:
- restarting, scaling, or failing over any production service
- rolling back a deploy
- rotating credentials or changing access
- anything touching a database or a VIP/revenue-critical service

High blast-radius actions require human approval even when you are confident.
Confidence does not lower the blast radius.

## When to escalate
If you cannot form a confident diagnosis, or the incident is P0/P1, call
`escalate` to page the on-call rotation while you keep gathering context.

Notice there are no efficiency targets in here. The agent's job is not to close incidents fast—it's to handle them correctly and hand off cleanly. Speed is a byproduct of removing the acknowledgement and context-gathering lag, not a goal we let it optimize against safety.

Step 2: Read-only diagnostic tools

These are the tools the agent can reach for without asking anyone, because they can't hurt anything. Create eve/agents/oncall/tools/get_alert_context.ts:

import { tool } from '@eve/core';
import { z } from 'zod';

export default tool({
  name: 'get_alert_context',
  description: 'Fetch the raw alert plus the service it belongs to, its owner, and its severity tier.',
  schema: z.object({
    alertId: z.string(),
  }),
  execute: async ({ alertId }) => {
    // In a real system, query your alerting provider (PagerDuty, Opsgenie, etc.)
    return {
      service: 'payments-api',
      tier: 'revenue-critical',   // <- feeds the blast-radius decision
      severity: 'P1',
      summary: 'p99 latency > 2s for 5m',
      firedAt: '2026-07-18T03:07:00Z',
    };
  },
});

That tier field is doing quiet but important work: it's the input the agent uses to reason about blast radius. A revenue-critical service means "diagnose freely, but do not touch without approval."

Now the diagnostic tool, eve/agents/oncall/tools/run_diagnostics.ts:

import { tool } from '@eve/core';
import { z } from 'zod';

export default tool({
  name: 'run_diagnostics',
  description: 'Pull recent logs, error rates, saturation metrics, and the last few deploys for a service. Read-only.',
  schema: z.object({
    service: z.string(),
    window: z.string().default('30m'),
  }),
  execute: async ({ service, window }) => {
    // Read-only queries to your observability stack.
    return {
      recentDeploys: [{ sha: 'a1b2c3d', at: '02:55Z', by: 'ci-bot' }],
      errorRate: 0.04,
      dbConnections: { used: 98, max: 100 },  // <- smoking gun
      hypothesis: 'connection pool exhaustion following 02:55 deploy',
    };
  },
});

The agent can call these as many times as it wants, in any order. Nothing here mutates production, so nothing here needs a gate.

Step 3: The approval gate for dangerous actions

This is the heart of the design. When the agent wants to do something that isn't safe-reversible-logged, it doesn't do it. It parks. Create eve/agents/oncall/tools/request_approval.ts:

import { tool, suspend } from '@eve/core';
import { z } from 'zod';

export default tool({
  name: 'request_approval',
  description: 'Request human sign-off for a high blast-radius action. Suspends the incident until an engineer approves or rejects.',
  schema: z.object({
    action: z.string(),          // e.g. "restart payments-api workers"
    reason: z.string(),          // the diagnosis that justifies it
    blastRadius: z.enum(['low', 'medium', 'high']),
  }),
  execute: async ({ action, reason, blastRadius }) => {
    // Notify the approver out-of-band (Slack, PagerDuty) with these details,
    // then suspend. The workflow state is checkpointed to durable storage and
    // the agent stops consuming resources until a decision arrives.
    return suspend({
      reason: 'awaiting_approval',
      message: `Approve action: ${action}\nWhy: ${reason}\nBlast radius: ${blastRadius}`,
      data: { action, blastRadius },
    });
  },
});

When this runs, the incident goes to sleep. Not a setTimeout, not a held-open request—the state is written to storage and the process is free to be restarted, deployed over, or scaled to zero. An engineer might approve in 30 seconds or 30 minutes. Either way, the agent resumes with everything it knew still in place. This is the same human-in-the-loop pattern we've covered before, applied where the stakes are highest.

The remediate tool then performs the action only after approval flows back through:

import { tool } from '@eve/core';
import { z } from 'zod';

export default tool({
  name: 'remediate',
  description: 'Execute an approved remediation. Must only be called after request_approval returns an approval.',
  schema: z.object({
    action: z.enum(['restart_workers', 'rollback_deploy', 'scale_up']),
    service: z.string(),
    approvalToken: z.string(),   // proof the human said yes
  }),
  execute: async ({ action, service, approvalToken }) => {
    if (!approvalToken) throw new Error('Refusing to remediate without approval.');
    // Perform the action against your infra API, and log it.
    return { performed: action, service, at: new Date().toISOString() };
  },
});

The approvalToken check is a belt-and-suspenders guard: even if the model somehow tried to skip the gate, the tool itself refuses to act without proof of sign-off. Encode your safety rules in the tools, not only in the prompt. Prompts can be argued with. A thrown error cannot.

Step 4: Escalation as a durable timer

Approval covers dangerous actions. Escalation covers the opposite problem: nobody's responding. For a P0, a common operational target is acknowledgement in under five minutes and mitigation in under fifteen. If the primary on-call doesn't ack, someone else needs to get paged—automatically.

The standard shape is a three-tier chain: primary on-call → backup on-call → engineering manager, where each tier gets auto-paged if the previous one doesn't acknowledge within a set window. That's a sequence of timed waits, which is exactly what durable execution is good at. Create eve/agents/oncall/tools/escalate.ts:

import { tool, sleep } from '@eve/core';
import { z } from 'zod';

const CHAIN = ['primary-oncall', 'backup-oncall', 'eng-manager'];

export default tool({
  name: 'escalate',
  description: 'Page the on-call chain for an incident, auto-advancing to the next tier if nobody acknowledges in time.',
  schema: z.object({
    incidentId: z.string(),
    severity: z.enum(['P0', 'P1', 'P2', 'P3']),
  }),
  execute: async ({ incidentId, severity }) => {
    const ackWindow = severity === 'P0' ? '5m' : '15m';

    for (const tier of CHAIN) {
      await page(tier, incidentId);      // your paging provider call
      await sleep(ackWindow);            // DURABLE wait — survives restarts

      if (await isAcknowledged(incidentId)) {
        return { acknowledgedBy: tier };
      }
    }
    return { acknowledgedBy: null, exhausted: true };
  },
});

sleep('5m') is not await new Promise(r => setTimeout(r, ...)). It's a durable timer. Eve.dev suspends the workflow and schedules a wake-up. Deploy your whole fleet during that five-minute window and the escalation still fires on time, because the timer lives in storage, not in a process's memory. That's the difference between an escalation policy you can trust and one that silently dies on your next release.

Step 5: Wire it to your alerting webhook

Now connect it to reality. Your alerting provider POSTs to a route; that route starts—or resumes—the incident. Create app/api/incident/route.ts:

import { Eve } from '@eve/core';
import { NextResponse } from 'next/server';

const eve = new Eve({
  workspace: './eve',
  storage: process.env.EVE_STORAGE_URL, // durable checkpoint store, e.g. Postgres
});

export async function POST(req: Request) {
  const event = await req.json();
  const agent = await eve.getAgent('oncall');

  // One incident per alert group. Eve.dev keys durable state on this id.
  const incidentId = event.groupKey;

  // If this incident is parked waiting for approval, `resume` carries the
  // human's decision back in. Otherwise it's a fresh alert and we start.
  const result = await agent.run({
    id: incidentId,
    input: event.approval
      ? undefined
      : `New alert: ${JSON.stringify(event.alert)}`,
    resume: event.approval, // { approvalToken, approved: true } or undefined
  });

  if (result.status === 'suspended') {
    return NextResponse.json({ state: 'awaiting_human', message: result.suspensionMessage });
  }
  return NextResponse.json({ state: 'resolved', summary: result.finalOutput });
}

Because Eve.dev keys everything on incidentId, the same endpoint handles both "an alert just fired" and "an engineer just clicked Approve in Slack." A fresh id starts a new incident; a known id that's parked resumes right where it suspended. You don't manage the state machine—you just hand Eve.dev an id and let it figure out whether this is a start or a continuation.

Step 6: Add a proactive health sweep on a schedule

So far the agent is reactive. But the fastest incident is the one you catch before the alert fires. Eve.dev can run an agent on a schedule, defined—of course—as a file. Create eve/agents/oncall/schedule.ts:

import { schedule } from '@eve/core';

export default schedule({
  cron: '*/5 * * * *', // every five minutes
  input: `Run a proactive health sweep. Check saturation and error-budget burn
          on revenue-critical services. If anything is trending toward an SLO
          breach, open a low-severity incident and gather context now — but do
          not take any action without approval.`,
});

The Eve runtime discovers this file and drives the agent every five minutes with a durable run of its own. Same rules apply: it can look at everything and touch nothing without a human. A scheduled sweep that quietly assembles context on a degrading service means that if it does tip into an incident, your agent already has the diagnosis half-written.

Testing and going to production

A few things that separate a demo from something you'd actually page:

Test the resume path, not just the happy path. The interesting bugs live in suspension. Write a test that starts an incident, drives it to awaiting_approval, then—critically—simulates a process restart before resuming. If your agent can't come back from cold storage and finish correctly, it isn't durable yet; it just hasn't been interrupted at the wrong moment.

Watch the acknowledgement gap, not just resolution time. "MTTR" is a famously slippery metric—people use it for mean time to respond, repair, recover, or resolve, and they rarely agree which. What's unambiguous is the decomposition: total time splits into detection, acknowledgement (the gap between the alert firing and a human actually starting work), and active repair. An on-call agent's real contribution is collapsing the acknowledgement-and-context-gathering portion toward zero. Measure that specifically, because that's the part you're changing.

Assume the diagnosis will sometimes be wrong. Everything in this design points back to one failure mode: a perfect action on a flawed hypothesis. That's why diagnostics are unlimited and free, actions are gated, and the gate can't be talked out of by a confident prompt. When you're deciding whether to let a new action run unattended, ask the blast-radius question, never the confidence question.

Keep a paper trail. Every approval, every action, every escalation is a checkpoint in Eve.dev's storage already. Surface it as an incident timeline. When the "continuous learning" stage rolls around—the blameless review after the dust settles—that timeline is your primary source, and it wrote itself.

Where this fits with the rest of your agents

This on-call agent leans on the same Eve.dev foundations as the other builds in this series, just pointed at higher stakes:

• The durable suspend()/resume mechanics come straight from the durable support agent tutorial.
• The approval gate is the human-in-the-loop pattern applied where a wrong automated action is most expensive.
• The file-per-capability layout follows the filesystem-first architecture that keeps larger agents readable.

Put together, you get a first responder that handles the boring 80% of an incident on its own, escalates the ambiguous cases with context attached, and—above all—never touches production on a hunch.

Frequently Asked Questions

Should an AI agent auto-remediate production incidents? Only the safe ones. The workable rule is to automate by blast radius and confidence together: low blast radius plus high confidence can run unattended, while anything high blast radius—databases, revenue-critical services, credential changes—should require human approval regardless of how confident the agent is. The reason is that a confidently wrong diagnosis produces a technically successful action aimed at the wrong target, and for high-impact actions that turns a small incident into a big one.

What's the difference between MTTA and MTTR? MTTA (mean time to acknowledge) measures the gap between an alert firing and a human actually beginning work on it. MTTR is broader and, frankly, ambiguous—it's used for mean time to respond, repair, recover, or resolve depending on who's talking. Total resolution time decomposes into detection time, acknowledgement time, and active repair time. An on-call agent mostly attacks the acknowledgement-and-context portion, so that's the number to watch.

How do you stop an AI agent from taking a dangerous action? Two layers. First, the instructions tell it which actions require approval. Second—and this is the one that actually holds—the tools themselves refuse to act without proof of sign-off. In this build, the remediate tool throws an error if it doesn't receive an approval token. Encoding safety in the tool means a persuasive prompt can't bypass it.

Can an Eve.dev agent survive a server restart mid-incident? Yes, and that's the main reason to use it here. Eve.dev checkpoints every tool call and state change to durable storage. When the agent waits for approval or on an escalation timer, the workflow is suspended to storage rather than held in memory, so a deploy, node drain, or crash doesn't lose the incident. When the trigger arrives, the agent resumes with full context.

How does the escalation timer work if the process gets redeployed? The sleep() primitive is a durable timer, not an in-memory setTimeout. Eve.dev suspends the workflow and schedules a wake-up in storage. If you redeploy during the wait window, the escalation still fires on schedule because the timer never depended on any single process staying alive.

What alerting tools does this work with? Anything that can send a webhook—PagerDuty, Opsgenie, Grafana alerting, and so on. The agent doesn't care about the source; the /api/incident route normalizes the payload and hands Eve.dev an incident id. The same route also receives approval callbacks from wherever your engineers click "approve," such as Slack.

Isn't a five-minute scheduled sweep expensive? It's a read-only run, so it's cheap relative to an outage. The sweep only gathers context and, at most, opens a low-severity incident—it never takes action on its own. The payoff is that when a slow degradation finally trips a real alert, the agent has already done the diagnostic legwork, which is the slow part of a 3 AM page.