Most founders who message me about AI don't actually want "AI." They want one specific thing to get smarter — search that understands intent, support that answers itself, a form that fills in the boring parts. Then somewhere between the idea and the budget, "add a feature" mutates into "rebuild the product around a model," and the whole thing stalls. This is the guide I wish they had: how to add AI to your existing web app without the rewrite.
It shouldn't be this hard. I've shipped AI into apps that were already live, already had paying users, and already had a schema I wasn't allowed to break. You don't need a rewrite. You need discipline about where the model goes and what it's allowed to touch. Here's how I approach it.
Start with a metric, not a model
The fastest way to waste money is to decide you're "adding AI" before deciding what it's supposed to change. "Add AI" isn't a spec. "Cut first-response time on support tickets in half" is. "Let users find a property by describing it in plain English instead of ten filters" is.
Pick the one feature where two things are true: the pain is obvious, and you already have good data sitting there. If you can't name the metric it moves, you're not ready to write code — you're ready to write a doc. That constraint alone kills 80% of the projects that would have failed anyway, and it's free.
Pick the smallest surface that touches the model
Once you know the metric, find the smallest possible slice of your app where the model does its job and hands control straight back. One endpoint. One input, one output. No sprawling "AI copilot" that reaches into every corner of the product on day one.
Small surface means you can reason about failure. When the model returns garbage — and it will, sometimes — you want exactly one place to look, one fallback to trigger, one thing to roll back. This is the same instinct behind good event-driven architecture: decouple the unpredictable thing so the rest of the system doesn't care when it misbehaves.
On ELD Trips Log Book, the intelligence lives in route and compliance generation — a bounded step that takes trip inputs and returns an optimized plan. The rest of the app is ordinary Django and Next.js. That boundary is why the smart part could evolve without threatening the parts drivers depend on to stay legal.
The orchestration layer is the only new architecture you need
You are not building a model. You're calling one. The single piece of new architecture worth adding is a thin orchestration layer that sits between your app and whichever provider you use.
That layer does four unglamorous jobs:
- Context building — assembling exactly what the model needs for this request, and nothing else.
- Routing — choosing the model and prompt template for the task (a cheap, fast model for classification; a stronger one for reasoning).
- Caching — returning stored answers for repeated or near-identical inputs.
- Fallback — a defined behavior for when the provider is slow, down, or returns nonsense.
Keep this in your own codebase, not scattered across React components. When a provider changes pricing or you want to swap models, you touch one module. Tool calling — letting the model trigger real functions in your app — lives here too, and the provider docs (see Anthropic's tool use guide) are worth reading before you invent your own protocol.
Retrieval is where quality is won or lost
The moment your feature needs to answer questions over your content — docs, listings, tickets, a knowledge base — you're doing retrieval, and retrieval is where most "the AI is dumb" complaints actually come from. It's rarely the model. It's that you fed it the wrong context.
I've written a whole post on why RAG pipelines quietly break, so I won't repeat it here. The short version: chunking, embedding quality, and what you retrieve matter far more than which model generates the final sentence. On Researcher, an app built entirely around AI-assisted research and writing, the difference between "useful" and "useless" was almost never the model choice — it was how deliberately the source material got prepared before it ever reached one.
Streaming and cost are product decisions
Two things founders treat as afterthoughts are actually product decisions you should make on purpose.
Streaming. A model that takes eight seconds to return a paragraph feels broken. The same model, streaming tokens as they generate, feels fast — because the user sees progress immediately. If your feature produces more than a sentence, stream it. This is a UX call, not a backend detail.
Cost. Output tokens usually cost several times more than input tokens, so response length is your biggest lever. Constrain outputs. Cache aggressively — semantically similar queries can reuse answers and cut spend dramatically in repetitive workloads. And log token usage per request from day one, or you'll get a bill you can't explain. A typical AI feature runs anywhere from $50 to a couple thousand dollars a month in API costs depending on volume; that number should never surprise you.
Data readiness is where projects actually die
Here's the unglamorous truth nobody sells you: the projects that stall don't stall on the model. They stall because the data the feature depends on is messy, incomplete, or locked in a format nobody can query.
Before you write the feature, audit the exact data it needs. Is it clean? Is it accessible? Is it complete enough that a smart intern could do the task by hand? If the answer is no, fix that first. A mediocre model on clean data beats a frontier model on garbage every single time.
The safest way to add AI to your existing web app
Put the feature behind a flag and roll it to a slice of users. Watch the metric you picked in step one. Watch cost. Watch the failure rate. Because you kept the surface small and the orchestration isolated, turning it off is a config change, not an incident.
Users don't want a new interface bolted onto your app. They want the interface they already use to quietly get smarter — the answer already there when they look, the boring field already filled. Add that in one place, measure it honestly, and expand only once it's earning its keep.
That's the whole method. No rewrite. One feature, one boundary, one metric — shipped, measured, and reversible. If you're staring at a live app wondering where AI fits, that's exactly where to start.