Putting a model in production

The Gap Between A Demo And A Feature

A model demo takes an afternoon. A feature built on a model takes considerably longer, and almost none of the extra time goes into prompting. It goes into deciding what happens when the output is wrong, because it will be, and a customer will be the one holding it.

In an editor a wrong answer costs you a few seconds. In a product it may cost a customer their afternoon or their trust. That difference should shape the design before a single call is made.

Where A Model Beats A Rule

The useful question is not "could a model do this" but "would a deterministic rule be worse". Models earn their place on problems that are genuinely ambiguous: pulling structure out of messy documents, classifying free text, summarising something a person would otherwise read in full.

They are a poor choice for anything with a correct answer you could compute. Whether a subscription has lapsed, whether a total adds up, whether a date falls inside a window. Putting a model there is an expensive way to make a reliable thing unpredictable.

Non-Determinism Is A Product Decision

The same input can produce different output on Tuesday than it did on Monday. That is not a bug to be engineered away, it is a property to be designed around.

Decide early whether your feature can tolerate it. A suggestion can vary. A calculation shown next to a currency symbol cannot. If the answer needs to be stable, cache the result against the input and treat regeneration as an explicit action rather than something that happens on every page load.

Put A Person In The Loop Where It Matters

The most reliable pattern we have found is unglamorous: automate the extraction, keep the approval. A model does the tedious part, a person confirms it, and nothing reaches a customer without that confirmation.

This is what makes the feature shippable at all, because it converts an unbounded risk into a review queue. It also produces something valuable almost for free: a growing set of corrected examples showing exactly where the model gets things wrong.

Cost, Latency And Failure

Three things that rarely appear in a prototype and always appear in production:

• Cost scales with use. A feature that is free during development can become the largest line on the infrastructure bill. Know the per-call cost and where the ceiling is.
• Latency is user-visible. Several seconds is normal. Design the interface for that honestly rather than hiding it behind a spinner that suggests something faster is coming.
• The provider will have a bad day. Every call needs a timeout, a retry policy that backs off, and a defined behaviour when the answer never arrives.

Evaluate Against Real Examples

Reading a handful of outputs and deciding they look right is not evaluation. Collect real inputs, including the ugly ones, write down what correct looks like for each, and run the set whenever the prompt or the model changes.

A few dozen cases in a file will catch the change that quietly broke a category of documents, which is the failure that otherwise reaches you as a confused support message weeks later.

Log Enough To Explain Yourself

When a customer asks why the system did something, "the model decided" is not an answer anyone accepts. Record the input, the output, the model and prompt version, and who approved it. Store it alongside the result rather than only in your logging platform.

This is what lets you answer a complaint, diagnose a regression, and demonstrate that a human signed off. It is also, increasingly, what an auditor will ask for.

Start Smaller Than You Want To

The most successful uses in real products are narrow. One task, clearly bounded, with a person reviewing the result and an obvious fallback when it fails.

That is far less exciting than the demo, and it is the version that is still running a year later.