AI that ships, not AI that demos

The AI feature demoed like a magic trick. One prompt on stage, a clean answer, the room leaning in. Then it shipped. Six days later a real user typed something slightly off, and the model confidently invented a refund policy that never existed.

I've seen this film enough times to call the ending from the opening credits. The demo ran on friendly inputs picked by the person who built it. Production is a stranger at 2am, typing in Hinglish with three typos, and the model has no idea it's wrong.

Every founder wants AI in the product. Fair. The demo is the easy part. The thing that survives real users is a different animal, and building that animal is most of the actual work.

The demo-to-production gap is where features go to die

A demo is a controlled shot. You know the input, you know the happy path, you run it until it looks clean. Production hands the model garbage, sarcasm, empty strings, and questions you never imagined. The model answers all of them in the same confident tone, right or wrong.

Demo vs shipped, side by side

Here's the split I keep in my head before I put a model anywhere near a product.

The right-hand column is the product. The left one is a party trick.

Measure it before you trust it. Evals before vibes.

Vibes are how most teams judge an AI feature. Someone tries it a few times, it feels good, ship it. That's not testing. That's a hunch wearing a lab coat.

An eval is a boring, beautiful thing: a set of real inputs paired with what a good answer looks like, run on every change, scored. It turns "feels smart" into a number that moves up or down when you touch the prompt. Skip it and you're flying blind while calling it intuition.

I keep a file of the ugliest inputs I can find. The angry ones. The ambiguous ones. The ones a competitor would screenshot. Every model swap and every prompt tweak runs against that set before anything else. Tooling like OpenAI's eval guide makes this cheap to stand up. The cost of skipping it shows up later, in public.

Guard it like any dependency that lies

An AI model is an unreliable service you call over the network. Treat it like one. You already wrap flaky APIs in timeouts and fallbacks. The model deserves the same suspicion, plus one extra worry the others don't have: it can hand you a confident answer that is completely made up.

So I validate the shape before I trust the content, set a timeout so a slow model can't freeze the screen, and always keep a boring fallback for when the smart path fails.

type Answer<T> = { data: T; source: "model" | "fallback" };
 
async function withGuardrails<T>(
  ask: () => Promise<unknown>,
  parse: (raw: unknown) => T | null,
  fallback: () => T,
  timeoutMs = 4000,
): Promise<Answer<T>> {
  const timeout = new Promise<never>((_, reject) =>
    setTimeout(() => reject(new Error("model timed out")), timeoutMs),
  );
 
  try {
    const raw = await Promise.race([ask(), timeout]);
    const clean = parse(raw);
    if (clean !== null) return { data: clean, source: "model" };
  } catch {
    // fall through to the safe path
  }
 
  return { data: fallback(), source: "fallback" };
}

The point isn't the exact code. It's the posture. The model gets to help. It never gets to be the only thing standing between your user and a broken screen.

Cost and latency are product decisions, not line items

A model call has two prices: rupees and seconds. Both are product constraints, and both get ignored until the bill or the loading spinner gets embarrassing.

The biggest model is rarely the right one. Plenty of the tasks I ship route to a smaller, faster model that costs a fraction and answers before the user blinks. I save the expensive model for the few calls that genuinely need the horsepower. Cache the repeats. Stream the long answers so the wait feels shorter than it is.

A feature that's brilliant and takes nine seconds loses to one that's good enough in one. Users don't grade on a curve.

Never hand the model the wheel on anything that matters

Human-led, AI-assisted. I say it a lot because it's the whole game. The model drafts, suggests, ranks, summarizes. A human approves anything that touches money, identity, or a promise to a customer.

This is where it ties back to something I wrote earlier: build for the buyer, not just the user. The buyer doesn't care that your AI dazzled in the boardroom. They care whether it's right at 2am when nobody's watching and their name is on the outcome. A human check on the risky path is how you earn that.

Put the human where the stakes are

The trick is placing the human where the cost lives, not everywhere. Review the refund, not the autocomplete. Judgment is expensive. Spend it where being wrong actually hurts, and let the model run free where the worst case is a shrug.

Frequently asked questions

Do you use AI to write your code?

Yes, as a power tool, not a co-founder. It speeds up the rough carpentry: boilerplate, first drafts, the tedious refactor nobody enjoys. I read every line before it ships, because the model doesn't know your business and will happily write something plausible and wrong. The judgment stays mine.

Which models do you use?

Whatever fits the job and the budget, and I'll tell you why. A cheap fast model for classification and short answers, a stronger one for reasoning that actually needs it. I don't marry a provider. I route each task to the smallest model that clears the bar, then measure to be sure it does.

How do you keep the costs sane?

Right-sizing the model, caching repeat calls, and setting hard limits so one runaway loop can't drain a month of budget overnight. I instrument spend from day one, not after the surprise invoice. Most AI cost problems are really "we used the biggest model for everything" problems.

How do you stop it hallucinating?

You can't fully, so you build like it will. Validation on the output, a fallback for when it fails the check, and a human on anything where a wrong answer costs real money. Grounding the model in your actual data helps a lot, and never letting it be the last word on the risky stuff helps more.

What actually ships

AI in a product isn't a garnish you sprinkle on top. It's an unreliable dependency you measure, guard, budget, and supervise like any other. Do that and you get something that works on a Tuesday at 2am, not just on stage in front of a friendly crowd. Skip it and you get applause that curdles into a support ticket by Friday.

If you want AI that does the work instead of the demo, that's the service I run. Tell me what you're building and I'll tell you honestly whether a model belongs in it. Sometimes the best answer is that it doesn't, and I'd rather say that now than after you've paid for it.