From a convincing demo to a system you can run.
Published 30 September 2026
A good demo is not a product
A demo works because the presenter chooses the input. A product meets documents nobody wrote for it, questions nobody anticipated and users who sometimes try to make it misbehave. The distance between the two is mostly engineering: defining the task, connecting the right data, measuring quality and controlling what the system can do.
This article lists what we think has to be true before an AI feature is put in front of real users. It is a checklist for judgment, not a guarantee, and not every item applies to every feature.
Define the task and the cost of being wrong
Start with what the system should do, in terms a non-technical colleague would accept, and what happens when it gets it wrong. A summary that is slightly off costs little. A wrong answer about a contract, a medical or financial matter, or an automated action in a customer's account can cost a great deal.
That cost decides the design. Low-cost tasks can run with light supervision. High-cost tasks need a person to approve the result, restricted permissions or a narrower scope. Writing this down early prevents the common failure of discovering the risk after launch.
Evaluate with your own cases
General benchmarks say how a model performs on someone else's questions. They do not say how your feature performs on yours. Build an evaluation set from real, representative cases: typical requests, difficult ones, and the ones that must fail safely, such as a question the data cannot answer or a request the user should not be allowed to make.
Run the set on every meaningful change: a new model version, a changed prompt, a different retrieval setting. Some judgments can be automated, others need human review, and it is fine to start small. What matters is that quality is measured rather than assumed, and that a regression is visible before users see it.
Ground the system and respect the data
Models are fluent whether or not they are right. Grounding narrows the gap by retrieving relevant information from your own sources and asking the model to answer from it, ideally with references the user can check. Retrieval quality then becomes part of the system's quality and needs to be evaluated in its own right.
Data handling needs explicit decisions: what is sent to which provider, what is stored and for how long, who can see which documents, and how all of this fits LGPD and your internal policies. Access rules must apply at retrieval time, so that the system never surfaces a document its user could not have opened directly.
Controls, oversight and operation
Production use needs the unglamorous parts: logging that lets you reconstruct what happened, monitoring for quality, latency and cost, limits that stop a runaway loop, and a fallback for when the model or provider is unavailable.
For agents that can act, not just answer, permissions deserve particular care. Give the system the least access that lets it do its job, require approval for actions that are costly or hard to undo, and relax that only as evaluation shows it is safe. Models and providers change quickly, so design the system so that the model can be replaced without rebuilding everything around it.
If you have a prototype and are unsure how far it is from production, an assessment can answer that in a bounded way, before more is invested.
