---
title: "Agent Washing: How to Tell a Real AI Agent From a Chatbot in a Vendor Demo"
description: "Gartner reckons only about 130 of the thousands of vendors selling agentic AI are the real thing. Here are the questions that separate an actual agent from a rebranded workflow tool, and what a truthful demo looks like."
image: "https://foundrysoft.co/api/og?type=article&title=Agent+Washing%3A+How+to+Tell+a+Real+AI+Agent+From+a+Chatbot+in+a+Vendor+Demo&cat=Business+%2F%2F+AI+Strategy&rt=11+min+read&au=Varun+Raj+Manoharan&dt=2026-08-17"
url: "https://foundrysoft.co/blog/agent-washing-vendor-demo-checklist"
---

Business // AI Strategy 2026-08-17 11 min read

# Agent Washing: How to Tell a Real AI Agent From a Chatbot in a Vendor Demo

Gartner reckons only about 130 of the thousands of vendors selling agentic AI are the real thing. Here are the questions that separate an actual agent from a rebranded workflow tool, and what a truthful demo looks like.

![Varun Raj Manoharan](https://foundrysoft.co/images/about/founder.webp)

Varun Raj Manoharan Founder & Principal Engineer

Agent Washing AI Vendor Evaluation Agentic AI AI Procurement Enterprise AI

## Key takeaways

-   Agent washing is the rebranding of chatbots, RPA, and workflow automation as agentic. Gartner estimates only around 130 of the thousands of vendors making the claim are genuine.
-   The fastest test is to ask the vendor to break their own demo live. Real agents recover or escalate. Scripted ones fall over in a way the sales engineer has to talk over.
-   Ask what happens on the run that fails. A vendor who cannot describe their failure rate is either hiding it or has never run at volume.
-   Being a workflow tool is not a crime. Buying one while paying agent prices for problems that need judgment is.

I sat in on a vendor evaluation earlier this year where the product did something genuinely impressive on stage and I could not work out, for about twenty minutes, whether any of it was real.

That uncertainty is the problem. The gap between a well-built demo of a scripted workflow and a well-built demo of an actual agent is close to zero, because both are designed to show the path that works. You only find out which one you bought later, when the inputs get weird.

Gartner has a term for this, agent washing, and an estimate that only about 130 of the thousands of vendors claiming agentic capability are actually delivering it. The rest are chatbots, robotic process automation, and workflow engines with new labels on the packaging.

Here is how I try to tell the difference, and what I have learned to ask.

## What actually makes something an agent

Skip the marketing definitions. In practice the distinction that matters commercially is this: can the system decide what to do next, at runtime, based on what it just found out?

A workflow tool has the path baked in. Step one, then step two, then a branch if a condition is met. The branches were written by a human in advance. It might use a language model inside a step, to classify or summarise, and that is fine and often the right architecture. But the shape of the run was decided before the run started.

An agent chooses its own sequence. It has a goal, a set of tools, and it works out the order. Give it a different input and it may take a path nobody anticipated, including a wrong one. That flexibility is the product and the risk in the same breath.

This distinction is not a purity test. It has a direct commercial consequence: if your problem has a knowable path, buy the workflow tool, because it is cheaper, faster, more predictable, and easier to audit. Pay for an agent when the path genuinely cannot be enumerated in advance, which is a real category but a much smaller one than the market implies.

## The demo questions that actually work

Vendor demos are rehearsed. You are not going to catch anything by watching. You catch things by asking for a deviation, live.

**Ask them to run it on your input, right now.** Not a sample you send ahead for them to test against. Something you bring to the call. The reaction tells you nearly everything. Confident vendors say yes and their thing does something reasonable or fails gracefully. The others explain why your data needs to be onboarded first, which is sometimes true and often the tell.

**Ask them to break it.** "Show me a run that fails." A real agent team has a stock of these and is not embarrassed by them, because they have spent months looking at failure modes and have opinions about them. The scripted product cannot easily produce a failure that is not just an error page.

**Ask what the agent does that you did not tell it to do.** This is my favourite question. Genuine agent builders light up, because the surprising behaviours are the interesting part of their week. They will tell you about the time it found a workaround they had not considered, or started doing something adjacent and unhelpful. A workflow vendor will say it does exactly what it is configured to do, which is a truthful answer to a question about a different product.

**Ask for the trace of a single run.** Not a dashboard. The actual step-by-step record of what it did: the reasoning, the tool calls, the intermediate results. If the system is real this exists, because you cannot build one without it. If what comes back is a status log with four rows and green ticks, you are looking at a workflow.

**Ask about the failure rate at volume.** Every real agent has one. Ask for the number on their largest deployment and how it is handled. "It doesn't really fail" is not a reassuring answer, it is a disqualifying one. The answer you want sounds like: about 6% of runs need human intervention, here is how we detect them, here is what happens next.

## The cost question they hope you skip

Ask what a completed task costs, not what a seat or a query costs.

Agents loop. Each iteration resends accumulated context, so a task that takes twelve steps does not cost twelve times one step, it costs considerably more. Failed runs that get retried cost twice or three times. A vendor pricing on completed outcomes has absorbed that risk and has thought hard about it. A vendor pricing per seat, in 2026, either has a workflow product or has not modelled their own unit economics.

That last point is worth dwelling on, because per-seat pricing has been falling out of the market for exactly this reason. When an agent works well, the buyer needs fewer seats, so seat pricing pays the vendor to under-deliver. Most serious vendors have moved to usage or hybrid models. Somebody still leading with per-seat for an agent product is telling you something about how agentic it really is.

Then ask what happens to your bill if usage triples. If the answer is a linear multiplication, fine. If they cannot answer, they have not run at that scale.

## Integration is where the real evaluation happens

Model capability stopped being the constraint a while ago. The most commonly cited blocker in enterprise agent deployments now is integration with existing systems, and that is where I would spend most of an evaluation.

Ask how the agent reaches your systems. If the answer is a set of prebuilt connectors, ask which of your specific systems are on the list and what happens for the ones that are not. If the answer involves the Model Context Protocol or a documented tool interface you can extend, that is a better sign, because it means you are not waiting on their roadmap for the integration you actually need.

Ask what identity the agent uses. The answer should be a scoped credential per agent, ideally short-lived, not a shared service account with broad access. This is the question that separates vendors who have been through an enterprise security review from those who have not, and it will save you a month later.

Ask what it can write to, and how a write gets approved. Reading is cheap to allow. Writing is where the incidents come from.

## The reference call is the real evaluation

Everything above is a filter. The decision gets made on references, and you have to ask for the right ones.

Ask for a customer in your industry, at your scale, who has been live for at least six months. Not a pilot. Live. The six-month mark is when the honeymoon has ended and the operational reality is known.

On that call, three questions do most of the work.

What surprised you after go-live? You will hear about the cost curve, the escalation volume, or the integration that took four times as long. All useful.

What percentage of runs need a human? Customers answer this honestly because they live it. Compare it to whatever the vendor told you.

What did you have to build yourselves? There is always something. The size of the something is the gap between the product and the pitch, and it belongs in your budget.

## When the workflow tool is the right answer

I want to be fair to the category being maligned here. A great deal of valuable automation does not need an agent, and pretending otherwise is how you end up with an unpredictable system doing a predictable job.

If the process has a fixed sequence, use a workflow engine and put a model inside the steps that need judgment. You get determinism, cheap debugging, and a system your compliance function will understand in one meeting.

The place agents earn their price is where the sequence depends on what is found along the way: research tasks, triage across messy inputs, anything where the next step is a function of the last result and the space of possible paths is too large to enumerate. That is a real and valuable category. It is just not every category, and the vendors know it, which is why the word gets stretched.

## A short version to take into the room

If you want the compressed test, it is three questions.

Show me a failed run and tell me your failure rate at volume.

Show me the full trace of a single run, step by step.

Run it on an input I brought with me, on this call.

A real agent product survives all three comfortably. I have watched two evaluations end in the first fifteen minutes on the third question alone, which was a good outcome for everybody except the vendor.

We build agentic systems for a living and we also get asked to sit in on evaluations of other people's, which is a strange and useful vantage point. If you have a vendor decision coming up, or you are trying to work out whether your problem even needs an agent, [we are happy to be a second pair of eyes.](https://foundrysoft.co/services)

#### Related reading

[AI Voice Agents Answer the Phone Now. What That Actually Does to Your Support Org.

Enterprise voice agents crossed from novelty to normal this year. The technology question is largely settled. The interesting questions are what happens to your escalation path, your metrics, and the people left holding the hard calls.

AI Voice Agents Customer Support Contact Center AI

](https://foundrysoft.co/blog/voice-agents-support-org-economics)[Shadow Agents: Somebody in Your Company Has Already Built One

Over a million agents have been built in Copilot Studio alone, most of them by people who are not engineers. Shadow AI is not a hypothetical governance risk, it is the current state of most enterprises. Here is how to find them and what to do that is not a ban.

Shadow AI AI Governance Agentic AI

](https://foundrysoft.co/blog/shadow-agents-ai-governance)[What Agentic AI Actually Looks Like in Supply Chain and Manufacturing Operations

Supply chain is the function where agents have the clearest business case and the worst data. Here are the four workflows where AI agents earn their keep in manufacturing and logistics, and the two where they consistently disappoint.

Agentic AI Supply Chain Manufacturing

](https://foundrysoft.co/blog/agentic-ai-supply-chain-operations)

#### Next Article

[

Why 40% of Agentic AI Projects Get Canceled, and What the Survivors Did Differently

](https://foundrysoft.co/blog/why-agentic-ai-projects-get-canceled)

Available for new projects

## Let's build something great.

Have a project in mind? We are an elite software and AI development studio ready to bring your ideas to production. Let's talk about your roadmap.

Start a Project [See our work](https://foundrysoft.co/work)

```json
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "FoundrySoft",
  "url": "https://foundrysoft.co",
  "logo": "https://foundrysoft.co/logo.svg",
  "description": "FoundrySoft builds production-grade software and AI systems for US companies, from an India-based team of senior engineers.",
  "sameAs": [
    "https://github.com/foundrysofthq",
    "https://www.linkedin.com/company/foundrysoft",
    "https://www.instagram.com/foundrysoft/"
  ]
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "WebSite",
  "name": "FoundrySoft",
  "url": "https://foundrysoft.co"
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "TechArticle",
  "headline": "Agent Washing: How to Tell a Real AI Agent From a Chatbot in a Vendor Demo",
  "description": "Gartner reckons only about 130 of the thousands of vendors selling agentic AI are the real thing. Here are the questions that separate an actual agent from a rebranded workflow tool, and what a truthful demo looks like.",
  "url": "https://foundrysoft.co/blog/agent-washing-vendor-demo-checklist",
  "mainEntityOfPage": "https://foundrysoft.co/blog/agent-washing-vendor-demo-checklist",
  "image": [
    "https://foundrysoft.co/images/blog/agent-washing-vendor-demo-checklist.webp"
  ],
  "datePublished": "2026-08-17",
  "dateModified": "2026-08-17",
  "keywords": "Agent Washing, AI Vendor Evaluation, Agentic AI, AI Procurement, Enterprise AI",
  "author": {
    "@type": "Person",
    "name": "Varun Raj Manoharan"
  },
  "publisher": {
    "@type": "Organization",
    "name": "FoundrySoft",
    "logo": {
      "@type": "ImageObject",
      "url": "https://foundrysoft.co/logo.svg"
    }
  }
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "BreadcrumbList",
  "itemListElement": [
    {
      "@type": "ListItem",
      "position": 1,
      "name": "Home",
      "item": "https://foundrysoft.co/"
    },
    {
      "@type": "ListItem",
      "position": 2,
      "name": "Blog",
      "item": "https://foundrysoft.co/blog"
    },
    {
      "@type": "ListItem",
      "position": 3,
      "name": "Agent Washing: How to Tell a Real AI Agent From a Chatbot in a Vendor Demo",
      "item": "https://foundrysoft.co/blog/agent-washing-vendor-demo-checklist"
    }
  ]
}
```