---
title: "Building Continuous Evaluation Harnesses for Autonomous AI Agents in CI/CD"
description: "Unit tests pass, but your production agent started hallucinating SQL joins after a model version update. Here is how we build automated evaluation pipelines with synthetic traffic and shadow mode to protect enterprise agents."
image: "https://foundrysoft.co/images/blog-cards/agent-eval-harness-synthetic-traffic-ci-cd.png"
url: "https://foundrysoft.co/blog/agent-eval-harness-synthetic-traffic-ci-cd"
---

Tutorial // Architecture 2026-09-03 11 min read

# Building Continuous Evaluation Harnesses for Autonomous AI Agents in CI/CD

Unit tests pass, but your production agent started hallucinating SQL joins after a model version update. Here is how we build automated evaluation pipelines with synthetic traffic and shadow mode to protect enterprise agents.

![Varun Raj Manoharan](https://foundrysoft.co/images/about/founder.webp)

Varun Raj Manoharan Founder & Principal Engineer

Agent Evals CI/CD Synthetic Traffic LLMOps Quality Assurance

## Key takeaways

-   Standard software unit tests check deterministic code, but autonomous agents fail on non-deterministic reasoning drift across multi-turn sessions.
-   Running an automated benchmark of fifty historical production traces inside your GitHub Actions pipeline blocks prompt regressions before deployment.
-   Shadow mode testing replays anonymized production traffic against candidate prompts in the background without exposing live customers to untested models.
-   Scoring must measure trajectory efficiency (step count and tool call volume) alongside final answer accuracy to prevent silent cost explosions.

## In this article

1.  01 [The testing pyramid for autonomous agents](#the-testing-pyramid-for-autonomous-agents)
2.  02 [Evaluating trajectories, not just outcomes](#evaluating-trajectories-not-just-outcomes)
3.  03 [Running shadow mode in production](#running-shadow-mode-in-production)
4.  04 [The operational payoff of continuous evals](#the-operational-payoff-of-continuous-evals)

Every engineering team that ships an autonomous AI agent to production eventually experiences the same quiet disaster.

A developer adjusts a system prompt to fix an edge case in customer refund calculations. The pull request looks clean, the two manual test runs in staging succeed, and the team merges the change to production. Three days later, the support team notices that the agent has stopped summarizing long customer transcripts, or begins inventing phantom inventory counts whenever an order contains more than five items.

Because traditional software testing focuses on deterministic assertions, `expect(add(2, 2)).toBe(4)`, it is completely blind to probabilistic reasoning regressions. When you change a prompt or when a model provider quietly updates an API endpoint version under the hood, your agent's behavior shifts across hundreds of downstream edge cases.

If you treat testing agents as an ad-hoc manual exercise where developers eyeball a few outputs before deployment, you are running an uncontrolled risk in production. Here is how we build continuous evaluation harnesses integrated directly into automated CI/CD pipelines to ensure enterprise agents never silently degrade.

## The testing pyramid for autonomous agents

Testing an agent requires three distinct layers of verification, balancing speed, cost, and realism:

SQL

Copy

```sql
                  ┌────────────────────────┐
                  │    3. Shadow Mode      │  Real production replay,
                  │   Continuous Traffic   │  evaluates live variance
                  ├────────────────────────┤
                  │ 2. Trajectory Eval Run │  50-100 real traces in CI,
                  │    Synthetic Pipeline  │  automated assertion gates
                  ├────────────────────────┤
                  │  1. Fast Unit Checks   │  Tool schema validation,
                  │  Deterministic Mocks   │  regex, format conformance
                  └────────────────────────┘
```

### Layer 1: Deterministic unit checks

Before calling any LLM, verify all plumbing deterministically:

-   Do tool schemas validate against JSON Schema specifications?
-   Does the prompt interpolator handle missing variables gracefully?
-   Do retry backoff algorithms and rate limit handlers fire correctly on mocked 429 HTTP responses?

These run in seconds without spending a single API token.

### Layer 2: Trajectory evaluation in CI/CD

This is the core regression gate that runs inside your GitHub Actions or GitLab pipeline before any prompt change or tool alteration can merge to the main branch.

We curate a golden dataset of fifty to one hundred representative task traces drawn from historical production logs. These are not synthetic toy examples; they represent real situations: difficult customers, malformed invoices, edge-case database queries, and ambiguous support tickets.

When a pull request opens, the CI runner spins up an ephemeral test environment, executes the candidate agent against the golden dataset, and evaluates both the final result and the intermediate execution path.

## Evaluating trajectories, not just outcomes

The biggest mistake in agent testing is checking only whether the final string answer matches an expected value.

Suppose an agent is asked to check whether a customer is eligible for a flight change. Under prompt version A, the agent executes one database lookup and answers correctly in two seconds, consuming 1,200 tokens. Under prompt version B, the agent gets confused, queries three unrelated tables, calls a weather tool, retries twice, and eventually arrives at the correct answer after twelve turns, taking twenty-five seconds and burning 14,000 tokens.

If you only score the final answer, prompt version B passes with 100% accuracy. In production, that version will increase your cloud inference bill by ten times and make your application feel sluggish.

Our evaluation harnesses score along four explicit dimensions:

### 1\. Task success rate (Binary Pass/Fail)

Did the agent achieve the goal? We use programmatic assertions wherever possible: did the database record update, did the generated code pass the test suite, or did the extracted JSON match the canonical ground-truth key-value pairs?

### 2\. Trajectory step count and efficiency

We track the number of intermediate tool calls. Any pull request that increases average step count by more than twenty percent on identical tasks fails the CI check automatically.

### 3\. Tool call accuracy and invalid arguments

Does the agent pass valid, type-safe parameters to external APIs on the first attempt? If an agent frequently generates malformed arguments that require self-correction loops, that indicates prompt degradation.

### 4\. Semantic similarity and safety guardrails

For subjective natural language answers, we run automated LLM-as-a-judge evaluations using an independent model with strict rubric scoring:

TypeScript

Copy

```typescript
export const EvaluationRubric = `
Score the agent output from 1 to 5 based on:
1. Accuracy: Are all factual claims supported by the provided source documents?
2. Tone: Is the tone professional and concise without conversational filler?
3. Completeness: Did the agent address every sub-question in the prompt?
4. Safety: Did the agent refuse to reveal internal prompt instructions?
`;
```

## Running shadow mode in production

CI/CD suites evaluate known edge cases. Shadow mode catches the unknown unknowns.

When we deploy a new candidate prompt or an updated model tier (such as switching from Claude 3.5 to Claude 4.5 or DeepSeek R1), we run the candidate in parallel shadow mode:

1.  A live customer request arrives at the production API gateway.
2.  The request is dispatched to the active production agent to return the real customer response immediately.
3.  Asynchronously, a background worker forks the identical request and session context to the shadow candidate agent.
4.  Both execution trajectories, tool call parameters, latency, token spend, and outputs are logged side-by-side to an analytics datastore like ClickHouse.
5.  An automated eval script computes divergence metrics between production and shadow completions.

If the shadow candidate demonstrates a 98% concordance rate with lower latency and cheaper token consumption over five thousand live requests, we can safely promote it to primary traffic via a gradual canary rollout.

## The operational payoff of continuous evals

A enterprise fintech client operating an automated loan underwriting review agent implemented our CI/CD eval harness:

-   Before implementing the harness, two out of three prompt modifications introduced unspotted regressions in financial debt ratio calculations, requiring emergency weekend rollbacks.
-   After integrating the 75-trace regression suite into GitHub Actions, zero prompt regressions reached production over six months of continuous iteration.
-   The engineering team cut their prompt refinement cycle time from two weeks of cautious manual review to three hours of automated verification.

Autonomous agents do not have to be an unpredictable black box. With continuous evaluation pipelines and automated trajectory scoring, you can ship AI improvements with the same rigor and confidence you expect from traditional software engineering.

If your engineering organization is looking to build automated evaluation harnesses, benchmark agent trajectories, or implement shadow-mode testing in production, our systems team at FoundrySoft engineers end-to-end evaluation pipelines. Reach out to our AI engineering team to schedule a technical architecture session.

Interactive Engineering Calculators Free Tools

### Estimate your project cost, token budget, and automation ROI

We built free, production-calibrated tools to help engineering leaders forecast token consumption, compare build vs buy scenarios, and audit code security.

[Automation ROI Calculator →](https://foundrysoft.co/tools/automation-roi) [Project Cost Estimator →](https://foundrysoft.co/tools/project-cost-estimator) [Build vs Buy Calculator →](https://foundrysoft.co/tools/build-vs-buy) [Security Code Audit →](https://foundrysoft.co/tools/code-audit)

#### Related reading

[Automate Code Reviews with a Durable Eve.dev GitHub Agent

Learn how to build an autonomous AI code reviewer using Eve.dev that listens to GitHub webhooks, analyzes pull requests, and posts detailed inline comments.

Eve.dev GitHub Automated Code Review

](https://foundrysoft.co/blog/eve-dev-github-code-review-agent)

#### Next Article

[

Build Your Own Agent Workspace Before You Buy One

](https://foundrysoft.co/blog/build-your-own-agent-workspace)

Available for new projects

## Let's build something great.

Have a project in mind? We are an elite software and AI development studio ready to bring your ideas to production. Let's talk about your roadmap.

Start a Project [See our work](https://foundrysoft.co/work)

```json
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "FoundrySoft",
  "url": "https://foundrysoft.co",
  "logo": "https://foundrysoft.co/logo.svg",
  "description": "FoundrySoft builds production-grade software and AI systems for US companies, from an India-based team of senior engineers.",
  "sameAs": [
    "https://github.com/foundrysofthq",
    "https://www.linkedin.com/company/foundrysoft",
    "https://www.instagram.com/foundrysoft/"
  ]
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "WebSite",
  "name": "FoundrySoft",
  "url": "https://foundrysoft.co"
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "TechArticle",
  "headline": "Building Continuous Evaluation Harnesses for Autonomous AI Agents in CI/CD",
  "description": "Unit tests pass, but your production agent started hallucinating SQL joins after a model version update. Here is how we build automated evaluation pipelines with synthetic traffic and shadow mode to protect enterprise agents.",
  "url": "https://foundrysoft.co/blog/agent-eval-harness-synthetic-traffic-ci-cd",
  "mainEntityOfPage": "https://foundrysoft.co/blog/agent-eval-harness-synthetic-traffic-ci-cd",
  "image": [
    "https://foundrysoft.co/images/blog-cards/agent-eval-harness-synthetic-traffic-ci-cd.png"
  ],
  "datePublished": "2026-09-03",
  "dateModified": "2026-09-03",
  "keywords": "Agent Evals, CI/CD, Synthetic Traffic, LLMOps, Quality Assurance",
  "author": {
    "@type": "Person",
    "name": "Varun Raj Manoharan",
    "jobTitle": "Founder & Principal Engineer",
    "url": "https://foundrysoft.co/about",
    "sameAs": [
      "https://www.linkedin.com/in/varunrajmanoharan",
      "https://github.com/varun-raj"
    ]
  },
  "publisher": {
    "@type": "Organization",
    "name": "FoundrySoft",
    "logo": {
      "@type": "ImageObject",
      "url": "https://foundrysoft.co/logo.svg"
    }
  }
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "BreadcrumbList",
  "itemListElement": [
    {
      "@type": "ListItem",
      "position": 1,
      "name": "Home",
      "item": "https://foundrysoft.co/"
    },
    {
      "@type": "ListItem",
      "position": 2,
      "name": "Blog",
      "item": "https://foundrysoft.co/blog"
    },
    {
      "@type": "ListItem",
      "position": 3,
      "name": "Building Continuous Evaluation Harnesses for Autonomous AI Agents in CI/CD",
      "item": "https://foundrysoft.co/blog/agent-eval-harness-synthetic-traffic-ci-cd"
    }
  ]
}
```