---
title: "Shipping Generative AI Features Without Breaking Production"
description: "A field guide to taking LLM-powered features from a flashy demo to a system you can trust on a Friday afternoon."
image: "https://foundrysoft.co/api/og?type=article&title=Shipping+Generative+AI+Features+Without+Breaking+Production&cat=Engineering+%2F%2F+LLMs&rt=8+min+read&au=Varun+Raj+Manoharan&dt=2026-06-18"
url: "https://foundrysoft.co/blog/shipping-generative-ai-features"
---

Engineering // LLMs 2026-06-18 8 min read

# Shipping Generative AI Features Without Breaking Production

A field guide to taking LLM-powered features from a flashy demo to a system you can trust on a Friday afternoon.

![Varun Raj Manoharan](https://foundrysoft.co/images/about/founder.webp)

Varun Raj Manoharan Founder & Principal Engineer

LLMs Production Architecture Evals

## Key takeaways

-   Version your prompts and review prompt changes in pull requests, because a one-word edit can break a feature like a bad commit.
-   Build an eval harness of 30 to 50 real inputs before the feature, and run it on every prompt or model change.
-   Assume the model will be wrong: use structured outputs, add a cheap validation check, and fall back to a human or safe default when confidence is low.
-   Track token usage from day one, cache aggressively, and route the easy 80 percent of requests to a smaller model.

### Overview

Demos are easy. A prompt, a model, a pretty UI, a room full of nodding heads, the whole thing wrapped up in twenty minutes and everyone leaves impressed. Then the applause dies down and the real work starts. Latency. Cost. The occasional hallucination that lands in front of an actual paying customer. And the general moodiness of these models, which is the part nobody warns you about until p99 latency triples on a Tuesday for no reason you can name.

We build this stuff for a living. After enough 2am pages you quit trusting the demo and start trusting a checklist instead. This is ours, more or less.

#### Treat the prompt as code, not a sticky note

Here's the mistake I see over and over. The prompt lives in a Slack thread someone pasted it into back in March, or it's a 600-token f-string buried three layers deep in a controller next to the retry logic. And that string is some of the highest-leverage logic in your whole system. Treat it like the rest of your logic.

Version it. Every edit becomes a deploy you can tag, diff, and roll back when it goes sideways. Pull the static instructions apart from the dynamic context you splice in at runtime, so you can actually reason about each piece without squinting. And, please, put prompt edits through the PR like anything else.

War story. We once shipped a "harmless" wording tweak that quietly knocked the model out of JSON mode, downstream parser swallowed the error and fell back to an empty object, and nothing caught it for a full day because the demo input happened to still parse. A one-word change can break a feature exactly as hard as a bad commit. It's just sneakier, because it doesn't look like code.

#### Build the eval harness before you build the feature

"It looked good when I tried it" is not a measurement. It's a vibe. Before you write a single line of the user-facing feature, write the thing that tells you whether the feature works at all.

Pull together 30 to 50 real inputs. Not the happy path you cherry-picked for the demo. The cursed ones: the empty string, the 4,000-word wall of text someone pasted, the input that's secretly in Portuguese, the prompt-injection attempt your intern thought was funny. Then decide what "correct" actually means per case, which is harder than it sounds. Sometimes it's an exact string match. Sometimes it's semantic distance against a reference. Sometimes you hand the output to a bigger model with a grading rubric and let it judge, which works better than I expected it to and worse than the papers claim. Run the whole set on every prompt edit, every model bump, every temperature change, and watch the trend line. Not the demo. The trend line.

> The teams that win at this aren't the ones with the cleverest prompts. They're the ones with the shortest gap between making a change and knowing whether it helped.

#### Assume the model will be wrong

The thing is probabilistic. So plan for the bad tail, not the comfy average.

Pin the output down wherever you can. Structured outputs and tool/function schemas turn "I really hope it returns valid JSON this time" into something close to a contract. Close. Then bolt a cheap check behind it, a tiny validator or a smaller model acting as a sanity gate, so the obvious garbage dies before any human lays eyes on it. And when confidence drops below whatever threshold you picked, don't guess. Hand off to a person, or fall back to a boring safe default. A confidently wrong answer is so much worse than no answer. Honestly I'd rather the feature say "I'm not sure" than make something up with conviction. Users forgive the first. They remember the second.

#### Watch the bill

A feature that burns, say, forty cents a call is a science experiment, not a product. (Pick your own scary number, the point holds.) Track token usage from day one, tagged by feature and route, or you'll be reverse-engineering the bill from logs three months in when finance asks. Cache hard, prompt prefixes especially. And route the easy 80 percent of traffic to a smaller cheaper model, keeping the flagship for the requests that genuinely earn it. Most requests don't.

#### The short version

Most of this is dull, unglamorous engineering pointed at one weird, twitchy component. Version the prompts. Measure constantly, not once. Assume the model will be wrong and build the scaffolding for when it is. Watch the bill. None of it is clever, and that's sort of the point. The clever part is real and genuinely fun, but it face-plants in production unless the boring stuff underneath holds. I still don't fully trust any of these systems, if I'm honest. I just trust the harness, the checks, and the rollback button. That's enough to ship on a Friday.

#### Related reading

[When Not to Build a Multi-Agent System

A multi-agent system buys you exactly one thing, context isolation, and most teams reach for one long before that trade-off is worth making.

Multi-Agent Systems AI Agents Agent Orchestration

](https://foundrysoft.co/blog/when-not-to-build-multi-agent-system)[How to Evaluate a New LLM in 48 Hours: The Shadow-Mode Playbook

A repeatable process for deciding whether a newly launched model belongs in your stack: shadow-mode traffic, the four metrics that matter, and why benchmark tables should never drive your routing table.

LLM Evaluation Evals Model Selection

](https://foundrysoft.co/blog/evaluate-new-llm-48-hours-shadow-mode)[Escaping LLM Vendor Lock-in: Migrating to Open-Source Models

Learn how to architect your AI applications to avoid LLM vendor lock-in. Discover strategies for migrating from OpenAI to self-hosted models like Llama 3.

AI Strategy Open Source Llama 3

](https://foundrysoft.co/blog/escaping-llm-vendor-lock-in)

#### Next Article

[

Build an AI Voice Agent with Twilio and a Realtime Model

](https://foundrysoft.co/blog/ai-voice-agent-twilio-realtime)

Available for new projects

## Let's build something great.

Have a project in mind? We are an elite software and AI development studio ready to bring your ideas to production. Let's talk about your roadmap.

Start a Project [See our work](https://foundrysoft.co/work)

```json
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "FoundrySoft",
  "url": "https://foundrysoft.co",
  "logo": "https://foundrysoft.co/logo.svg",
  "description": "FoundrySoft builds production-grade software and AI systems for US companies, from an India-based team of senior engineers.",
  "sameAs": [
    "https://github.com/foundrysofthq",
    "https://www.linkedin.com/company/foundrysoft",
    "https://www.instagram.com/foundrysoft/"
  ]
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "WebSite",
  "name": "FoundrySoft",
  "url": "https://foundrysoft.co"
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "TechArticle",
  "headline": "Shipping Generative AI Features Without Breaking Production",
  "description": "A field guide to taking LLM-powered features from a flashy demo to a system you can trust on a Friday afternoon.",
  "url": "https://foundrysoft.co/blog/shipping-generative-ai-features",
  "mainEntityOfPage": "https://foundrysoft.co/blog/shipping-generative-ai-features",
  "image": [
    "https://foundrysoft.co/images/blog/shipping-generative-ai-features.webp"
  ],
  "datePublished": "2026-06-18",
  "dateModified": "2026-06-18",
  "keywords": "LLMs, Production, Architecture, Evals",
  "author": {
    "@type": "Person",
    "name": "Varun Raj Manoharan"
  },
  "publisher": {
    "@type": "Organization",
    "name": "FoundrySoft",
    "logo": {
      "@type": "ImageObject",
      "url": "https://foundrysoft.co/logo.svg"
    }
  }
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "BreadcrumbList",
  "itemListElement": [
    {
      "@type": "ListItem",
      "position": 1,
      "name": "Home",
      "item": "https://foundrysoft.co/"
    },
    {
      "@type": "ListItem",
      "position": 2,
      "name": "Blog",
      "item": "https://foundrysoft.co/blog"
    },
    {
      "@type": "ListItem",
      "position": 3,
      "name": "Shipping Generative AI Features Without Breaking Production",
      "item": "https://foundrysoft.co/blog/shipping-generative-ai-features"
    }
  ]
}
```