Getting an LLM POC to production: the path, in order

The path from a working LLM prototype to production is a sequence, and it is almost never “make the model better.” It’s: decide what good enough means, build an eval harness, get a reproducible deploy path, add observability and guardrails, put a ceiling on cost, and name an owner — roughly in that order. The order matters because each step makes the next one safe. This is the practical companion to The Production-Readiness Bar: the bar tells you what is missing; this tells you in what order to close it.

Start by deciding what “good enough” means

Before any pipeline work, answer two questions: what does an acceptable output look like, and what does a wrong one cost? A summarizer that occasionally drops a bullet is a different risk than an agent that can issue a refund or disclose the wrong patient’s record. That answer sets two things at once — your evaluation target (the bar the system has to clear) and your guardrail budget (how much you invest in controlling outputs). Teams that skip this step productionize a moving target: every change feels dangerous because nothing defines “working.” Pin it down first, in writing.

The sequence

Work top to bottom. Each row assumes the ones above it are in place — that dependency is the whole point.

Order	Step	You’re done when	Depends on
1	Define the target	You have a written pass bar and the cost of a bad output	—
2	Eval harness	A scored, repeatable pass/fail runs on a representative dataset	1
3	Reproducible deploy	Any change ships via a gated, rollback-safe pipeline — no click-ops	2
4	Observability	You can trace what was retrieved, prompted, returned, and acted on	3
5	Guardrails	Inputs/outputs are controlled and high-consequence actions gate on a human	1, 4
6	Cost ceiling	Spend is attributable per feature and a cap pages before it bankrupts	3
7	Named owner	A specific human runs it, watches it, and is on call	3–6

Why evals come before deployment

A deploy pipeline is only as useful as your ability to know whether the thing you just shipped is better or worse. Without evals, “reproducible deploy” just means you can reliably ship regressions. The eval harness is what converts a deploy from a held breath into a gated, scored decision — so build it first, then wire it into the pipeline as the gate that blocks a bad change.

Why observability comes before you scale traffic

The default state of an unobserved LLM system is silent failure. Before you widen the audience, you need traces: what the model retrieved, what it was prompted with, what it returned, and what action it took. Otherwise your first real incident is also the first time you try to reconstruct what happened — by re-running it and hoping. Observability is cheap to add before launch and expensive to retrofit after one.

Why guardrails are sized to blast radius, not to fear

Guardrails are input/output controls plus human-in-the-loop on the actions that carry real consequences. Size them to the cost you wrote down in step 1. A read-only assistant needs light output filtering; an agent that can move money or touch PHI needs policy-as-code and an approval gate on the consequential action. Over-guarding a low-risk feature wastes weeks; under-guarding a high-risk one fails security review and kills the launch.

What usually goes wrong

• The rewrite-the-model trap. The prototype stalls, so the team assumes it needs a better model or more prompt engineering. Almost always the blocker is one of steps 2–7, and more model work doesn’t touch it.
• Big-bang launch. Shipping to all users at once, with no eval gate and no traces, means the first regression is a public one. Ship behind a flag, to a slice, with observability on.
• Evals as an afterthought. “We’ll add evals once it’s live” means quality stays a matter of who last clicked through a demo. The harness is step 2 for a reason.
• No owner, or “the AI team owns it.” That means no one is on call. An unowned model in production is a liability the moment the person who understood it moves on.

How long it takes

The honest answer: it depends on how many bars are already met and your regulatory posture, not on model work. A prototype with a clean deploy path and no sensitive data moves quickly; one with PHI in its logs, no evals, and click-ops deployment does not. What’s predictable is the order — the steps don’t reshuffle, so you can scope the work as soon as you know which ones are missing.

That scoping is exactly the production-readiness audit: a fixed-scope assessment against this bar, a prioritized risk register, and a scoped path to close the gaps — run by the person who’d do the buildout. You can take the findings and walk, or have the same hands close them.