Most AI prototypes never become production systems. The ones that do often follow a predictable arc: a demo impresses stakeholders, the team gets approval to ship, the model is wrapped in an API, and within 90 days there is an incident. Not because the model was wrong -- the model may have been performing exactly as it did in the demo. The incident happens because nobody built the governance layer. Nobody asked what happens when the model makes a decision that needs to be explained, reversed, or audited. Nobody defined who gets notified when the system encounters something it was not designed to handle.
The gap between a working prototype and a production-grade AI system is not primarily a model quality gap. It is a governance gap. Stanford HAI's research on responsible AI deployment identifies this pattern repeatedly: teams optimize for model performance during development and discover governance requirements -- logging, policy enforcement, human oversight, change management -- only after production failures force the conversation.
This article provides a concrete checklist of governance capabilities that must exist before an AI system takes consequential actions in production. Each item is explained with the specific failure mode it prevents. The checklist is deliberately tool-agnostic -- these capabilities must exist regardless of what model, framework, or platform you use. Skipping any item does not mean the system will fail immediately. It means the system will fail in a way that is harder to diagnose, slower to remediate, and more expensive to explain.
Why Prototype-to-Production Fails at Governance
The prototype phase optimizes for a single question: does the model produce good outputs? This is the right question for a prototype. It becomes the wrong question when the system is processing real decisions for real users under real regulatory and business constraints.
In production, the questions that matter are different. Can you reconstruct why the system made a specific decision last Tuesday? Can you demonstrate that the rules governing the system today are the same rules that were governing it when that decision was made? If a rule needs to change, can you change it without redeploying the entire system? If something goes wrong, can you stop the system from making more of the same decision within minutes, not hours?
The NIST AI Risk Management Framework structures these concerns across its Map and Measure functions. The Map function requires organizations to identify the context in which the AI system operates, including the potential impacts of its decisions. The Measure function requires that those impacts be quantifiable and monitored. Together, they describe the governance surface that a production AI system must expose -- and that most prototypes do not.
The checklist that follows is organized around six governance capabilities. Each capability addresses a category of production failure. None of them are about model quality.
Checklist Item 1: Decision Logging Schema
What must exist
Every consequential decision the AI system makes must be logged in a structured, queryable format. The log must capture: the inputs that were evaluated, the decision that was made, the rule or policy that governed the decision, the version of that rule at the time of evaluation, the timestamp, and the outcome (what happened as a result of the decision).
# Minimum decision log schema
decision_record:
id: "dec-20260615-00847"
timestamp: "2026-06-15T14:23:07Z"
system: "expense-approval-agent"
inputs:
request_id: "exp-44291"
amount_usd: 7200
category: "consulting"
requester: "user-8812"
decision: "deny"
governing_rule:
name: "consulting-spend-limit"
version: "2.1.0"
condition: "amount_usd > 5000 AND category = consulting"
outcome:
action_taken: "denial_notification_sent"
escalation_path: "manager-approval-queue"
trace_id: "tr-exp-20260615-00847"What failure looks like without it
A customer or stakeholder asks: "Why was my expense denied?" Your team searches application logs for the timestamp, finds a model inference record that shows input tokens and output tokens, and pieces together what probably happened from log fragments across three services. The reconstruction takes hours and produces an approximation, not an authoritative answer. If a regulator asks the same question, the approximation is not sufficient.
Google's Responsible AI Practices emphasize that systems should be designed to explain their behavior in ways that are meaningful to stakeholders. A decision log schema is the infrastructure that makes this possible. Without it, explainability is aspirational documentation rather than an operational capability.
Checklist Item 2: Policy Version Control
What must exist
The rules and policies that govern AI decisions must be versioned independently of the application code. Each version must be an immutable snapshot: once a version is created, it cannot be modified, only superseded by a new version. The system must be able to answer, at any point in time: which version of which rules was active when a specific decision was made?
Policy version control requires three properties that are distinct from code version control:
- Immutable snapshots: Each policy version is preserved exactly as it existed when it was active. No retroactive edits. No in-place modifications.
- Temporal querying: Given a timestamp, the system can return the exact set of policy rules that were active at that moment. This is not the same as looking at the current rules -- it requires a point-in-time retrieval capability.
- Change lineage: Each policy version records why it was created, who authored it, who approved it, and what version it replaced. The lineage chain is unbroken from the current version back to the first version.
What failure looks like without it
A rule change is deployed to production. Two weeks later, an audit reveals that several decisions made under the previous rule version produced outcomes that the new rule would have prevented. The team needs to identify exactly when the rule changed and which decisions were made under each version. Without policy version control, the team cannot answer this question definitively. They reconstruct the timeline from deployment logs, git commits, and team memory -- producing an estimate, not a fact. If the rule was embedded in a prompt template or application configuration, the previous version may have been overwritten entirely.
Checklist Item 3: Rollback Procedures
What must exist
The team must be able to revert any policy rule to a previous version within minutes, without redeploying the application. Rollback must be an atomic operation: the previous version becomes active, the current version is archived, and all subsequent decisions are evaluated under the restored version. The rollback itself must be logged as a governance event.
Rollback requires that the system maintain a ready-to-activate copy of every previous policy version. It also requires that the rollback operation handle in-flight decisions correctly: decisions that are currently being evaluated when the rollback occurs must either complete under the old version or be re-evaluated under the restored version, with the handling explicitly defined rather than left to chance.
# Rollback operation record
rollback:
id: "rb-20260615-001"
timestamp: "2026-06-15T16:45:00Z"
initiated_by: "ops-lead-jchen"
reason: "Consulting spend limit denying pre-approved vendor renewals"
rule_rolled_back:
name: "consulting-spend-limit"
from_version: "2.1.0"
to_version: "2.0.0"
in_flight_handling: "re-evaluate-under-restored-version"
decisions_affected:
pending_re_evaluation: 3
completed_under_old_version: 47
rollback_duration_seconds: 4What failure looks like without it
A policy change causes unexpected denials in production. The team identifies the problem within an hour. The fix requires modifying the rule, testing it, and redeploying the application. The redeployment takes 45 minutes. During that time, the faulty rule continues to make incorrect decisions. If the rule is embedded in application code and requires a full CI/CD pipeline to change, the window of incorrect decisions can extend to hours. Every incorrect decision during that window becomes a remediation task and, potentially, a compliance incident.
Checklist Item 4: Human Escalation Paths
What must exist
Every decision category must have a defined escalation path: what happens when the AI system encounters a situation it is not designed to handle, when a decision is contested, or when the system's confidence falls below a defined threshold. The escalation path must specify who receives the escalation, what information they receive, what actions they can take, and what the timeout behavior is if no human responds.
Escalation paths must be tested in the same way that deny paths are tested. An escalation that routes to a Slack channel that nobody monitors is not an escalation path -- it is a dead letter queue with a notification icon. An escalation that sends a human reviewer 40 pages of model context with no summary is not actionable -- it is a data dump disguised as a request for judgment.
| Escalation Trigger | Routed To | Information Provided | Timeout Behavior |
|---|---|---|---|
| Decision confidence below threshold | Domain specialist queue | Input summary, model recommendation, confidence score, relevant policy | Auto-deny after 4 hours with notification |
| Contested decision (user appeal) | Senior reviewer | Original decision trace, user appeal text, full input record | Escalate to manager after 24 hours |
| Policy conflict (multiple rules disagree) | Policy owner | Conflicting rule versions, input that triggered conflict, both proposed outcomes | Apply most restrictive outcome after 1 hour |
| Out-of-scope input (no matching rule) | Operations team | Input record, list of evaluated rules, reason for non-match | Auto-deny with reason "requires manual review" |
What failure looks like without it
The AI system encounters an input it was not designed to handle -- a request type that did not exist in the training data, or a combination of parameters that no rule covers. Without a defined escalation path, the system either makes a best-guess decision (which may be wrong and will not be flagged for review) or throws an error that propagates to the user as a generic failure message. In both cases, nobody with domain expertise is notified, and the decision either goes unreviewed or the user experiences an unexplained failure. The Thoughtworks Technology Radar 2025 flags the absence of human-in-the-loop escalation as a leading indicator of AI production incidents in enterprise deployments.
Checklist Item 5: Drift Detection
What must exist
The system must continuously monitor for two types of drift: input drift (the distribution of inputs the system receives is changing in ways that may affect decision quality) and decision drift (the distribution of decisions the system makes is changing in ways that may indicate a problem).
Input drift detection monitors the statistical properties of incoming requests. If the system was designed to handle expense approvals between $100 and $50,000 and begins receiving requests for $500,000, that is input drift. If the category distribution shifts from 60% travel / 30% consulting / 10% equipment to 20% travel / 70% consulting / 10% equipment, that is input drift. The model may still produce outputs for these shifted inputs, but the outputs may not be reliable because the inputs have moved outside the distribution the system was validated against.
Decision drift detection monitors the outcomes the system produces. If the approval rate drops from 85% to 60% over two weeks without a corresponding policy change, something has changed -- either the inputs shifted, the model behavior shifted, or an interaction between rules is producing a compounding effect. Decision drift detection catches this before the cumulative impact becomes a business problem.
# Drift detection alert configuration
drift_monitors:
- name: "input-amount-distribution"
metric: "expense_amount_usd"
baseline_period: "30d"
detection_method: "kolmogorov_smirnov"
alert_threshold: 0.05
alert_channel: "ops-alerts"
- name: "decision-approval-rate"
metric: "approval_rate"
baseline_period: "14d"
detection_method: "percentage_change"
alert_threshold: 0.10
alert_channel: "policy-owners"
- name: "escalation-volume"
metric: "escalations_per_day"
baseline_period: "7d"
detection_method: "absolute_threshold"
alert_threshold: 25
alert_channel: "ops-alerts"What failure looks like without it
The system operates correctly at launch. Over four months, the input distribution shifts gradually. The model continues to produce outputs, but its decision quality degrades because the inputs no longer resemble what the system was validated against. The degradation is invisible because nobody is monitoring for it. The problem surfaces when a quarterly review reveals that decision accuracy has dropped significantly -- but the degradation happened incrementally, across thousands of decisions, and cannot be attributed to a single event. The remediation requires re-validating the system against the current input distribution and potentially retraining or adjusting rules -- work that could have been triggered weeks earlier if drift had been detected.
Checklist Item 6: Audit Report Generation
What must exist
The system must be able to produce, on demand, a complete governance report for any time period. The report must include: the total number of decisions made, the breakdown by decision type and outcome, the policy versions active during the period, any policy changes that occurred during the period (with change rationale and approval records), any escalations and their resolutions, any drift alerts and the responses to them, and any rollbacks performed.
The report must be producible without manual assembly. A governance report that requires an engineer to query three databases, join the results in a spreadsheet, and format the output over the course of two days is not an audit report -- it is a research project. The report generation capability should be automated to the point where producing a report for any time period takes minutes, not days.
The NIST AI RMF Measure function explicitly requires organizations to track and document AI system performance and risk indicators over time. The audit report is the operational artifact that satisfies this requirement. Without automated generation, the cost of producing audit reports creates pressure to produce them less frequently, which creates longer gaps between governance reviews, which increases the risk that problems go undetected.
What failure looks like without it
A compliance team requests documentation of all AI-assisted decisions made in Q1. The engineering team begins assembling the report. They discover that decision logs are spread across three services with different schemas. Policy version history requires cross-referencing git commits with deployment timestamps. Escalation records are in a ticketing system that is not connected to the decision logging infrastructure. The report takes two weeks to assemble and contains gaps where data was not captured or was captured in an incompatible format. The compliance team receives the report too late to include in their quarterly filing, and the gaps reduce confidence in the AI system's governance posture.
The Complete Checklist
The following table summarizes the six checklist items with their governance function and the failure mode they prevent:
| Checklist Item | Governance Function | Failure Prevented |
|---|---|---|
| Decision logging schema | Traceability | Inability to explain or reconstruct past decisions |
| Policy version control | Accountability | Cannot determine which rules governed a specific past decision |
| Rollback procedures | Incident response | Extended window of incorrect decisions during remediation |
| Human escalation paths | Human oversight | System makes decisions on inputs it was not designed to handle |
| Drift detection | Continuous validation | Gradual decision quality degradation goes undetected for weeks or months |
| Audit report generation | Compliance readiness | Governance documentation requires weeks of manual assembly and contains gaps |
Sequencing the Checklist: What to Build First
Teams moving from prototype to production rarely have the capacity to implement all six items simultaneously. The following sequencing reflects the dependency chain between items and the urgency of each failure mode:
Phase 1: Decision logging schema and human escalation paths. These are the foundation. Without decision logging, none of the other items can function -- policy version control, drift detection, and audit reports all depend on structured decision records. Without escalation paths, the system is fully autonomous with no safety valve, which is the highest-risk configuration for a production AI system.
Phase 2: Policy version control and rollback procedures. Once decisions are being logged, the next priority is ensuring that the rules governing those decisions are versioned and reversible. This is the difference between "we can see what happened" (Phase 1) and "we can control and correct what happens" (Phase 2).
Phase 3: Drift detection and audit report generation. These are monitoring and reporting capabilities that build on the infrastructure established in Phases 1 and 2. Drift detection requires baseline decision data (from Phase 1 logging). Audit reports require policy version history (from Phase 2 version control) combined with decision records (from Phase 1 logging).
This sequencing can typically be executed in 8 to 12 weeks for a single AI system. Teams that attempt to build all six capabilities in the last two weeks before launch -- which is common when governance is treated as a launch checklist rather than an infrastructure requirement -- typically ship with significant gaps in at least two items.
The Underlying Principle: Governance Is Infrastructure, Not a Gate
The most common reason teams skip governance items is that governance is framed as a gate: a set of requirements that must be satisfied before launch. Gates create pressure to minimize the requirements so the launch can proceed on schedule. Governance items are deferred, scoped down, or replaced with documentation that describes what the team intends to build later.
The alternative framing is governance as infrastructure: capabilities that the production system depends on in the same way it depends on logging, monitoring, and deployment automation. You would not ship a production system without application logs. You would not ship a production system without the ability to roll back a bad deployment. The governance capabilities in this checklist are the AI-specific equivalents of those baseline expectations.
When governance is infrastructure, it is built alongside the system rather than bolted on at the end. The decision logging schema is designed when the decision logic is designed. The escalation paths are defined when the decision categories are defined. The rollback procedures are tested when the policy rules are tested. The work happens in parallel with development, not after it.
Teams that treat governance as infrastructure report a consistent outcome: the governance layer catches problems that would otherwise have become production incidents. A rule that would have been deployed directly to production instead goes through shadow evaluation and reveals an unintended interaction. An escalation path that would have been undefined instead routes a novel input to a domain expert who catches an error before it reaches a customer. A drift monitor that would not have existed instead triggers an alert three weeks before a quarterly review would have revealed the same problem.
None of these outcomes are dramatic. They are the absence of drama -- which is what production systems are supposed to provide.