What an AI Feature Needs Before It Ships: Guardrails, Evaluation, and a Human in the Loop

#Why a good demo is not the same as a shippable feature

A demo is a curated set of inputs shown to people who mostly want it to work. Production is an open set of inputs from people who did not choose them to flatter the system. The gap between the two is exactly where AI features fail after launch: the model handles the expected cases well and handles an unexpected input by producing something confident, plausible, and wrong, with nothing in the system built to catch it.

#Guardrails: constraining what the system can do, not just what it says

A guardrail is a constraint placed around the model's output or its actions, independent of the model's own judgment. If the feature drafts an email, a guardrail can require a human to send it rather than letting the system send automatically. If the feature classifies a support ticket, a guardrail can cap how confidently it acts on a low-confidence classification, routing anything below a threshold to a person instead of auto-resolving it. Guardrails do not make the model smarter. They limit the blast radius when it is wrong, which every model is, some percentage of the time, regardless of how good its average performance looks.

#Evaluation: measured on an ongoing basis, not once before launch

A single evaluation run before launch tells you how the system performed against the inputs you tested it with. It does not tell you how it performs six weeks later, once real users send inputs nobody anticipated, or once the underlying model is updated by its provider and its behaviour shifts in ways that were never announced as a breaking change. An evaluation process that keeps sampling real outputs, scoring them against a rubric, and flagging drift is what makes it possible to catch a quality regression before a client does, rather than after.

#Human in the loop: placed where the cost of a wrong answer is highest

Not every output needs a human to review it before it goes out, and treating every single output as needing review defeats the purpose of automating the work in the first place. The useful design decision is where the review point goes: usually wherever a wrong output is expensive to reverse, carries legal or financial weight, or would damage a relationship if it reached a client unreviewed. A low-stakes internal draft can go straight through. A client-facing communication or a financial decision usually should not, at least not without a clear, fast review step built into the workflow rather than added as an afterthought once something has already gone wrong.

#A framework that already covers this ground

None of this is a novel argument. The NIST AI Risk Management Framework, a voluntary framework published for organizations building and deploying AI systems, structures exactly this territory into four core functions: governing how the system is built, mapping where risks could arise, measuring how it actually performs, and managing the risks that measurement surfaces. Guardrails, ongoing evaluation, and a defined human review point map directly onto the "measure" and "manage" functions in that framework, which is one of the reasons we treat them as a checklist item before launch rather than a nice-to-have added once something breaks.

#The actual pre-launch checklist

Before a feature backed by a model ships, we want a written answer to four questions: what happens when the model produces an output outside its expected range, who reviews outputs before high-stakes consequences attach to them, how often is quality re-measured after launch and against what rubric, and what is the rollback plan if a quality regression is detected. A feature that can answer all four is ready. A feature that can only point to a good demo is not, no matter how good that demo looked.