An AI agent governance framework is often described as a set of policies: who can build agents, what they can do, and how they are reviewed. That framing is incomplete. A document on a shelf cannot pause a runaway tool call, verify that an approved payload matches what actually executed, or alert a team when a cost anomaly signals a loop. Those functions require an operating system that connects authorization, observability, evidence, release gates, ownership, and cost visibility into a single control surface.
The difference matters because production AI agents act across systems that were never designed for agent-driven requests. A customer-service agent that reads a policy, updates a case, and sends a message touches identity, CRM, and communications infrastructure in one flow. An internal knowledge agent may retrieve data, generate a summary, and write back to a knowledge base. Each step introduces a governance question: was the action authorised, was the evidence current, was the execution correct, and can the organisation verify all three after the fact?
This guide treats governance as an executable control system — not a compliance exercise. It connects the operational controls that production teams already need with the oversight obligations that regulators, auditors, and business leaders expect.
Governance Starts with Action Classification and Authority Tiers
A governance framework must first answer a concrete question: what types of actions can this agent take, and who is allowed to decide? The answer is not a single policy line. It is a classification of every action the agent can touch, assigned to an operating tier.
A practical tier model has four levels:
- Auto-execute: low-consequence, reversible, well-tested actions where the agent acts without human involvement. Example: drafting a reply from an approved template.
- Execute and notify: actions that proceed automatically but generate an asynchronous record for sampling or review. Useful when the action is low risk but a human should know it happened.
- Approve before execute: consequential actions that pause the workflow, assemble an approval packet, route it to an authorised reviewer, and resume only on a valid decision. These are the actions covered in our detailed guide on designing an AI agent approval workflow.
- Prohibit: actions the agent must never perform, even with ad hoc approval. Escalating a prohibited action triggers a separate security or policy review before any configuration change is made.
The classification should use objective dimensions: reversibility, blast radius, data sensitivity, external commitment, privilege level, and evidence quality. Model confidence alone is insufficient — a highly confident agent can still apply a policy to the wrong customer or use an outdated source.
OWASP's guidance on excessive agency reinforces the security case for least privilege enforcement and narrow action boundaries. Every tier above auto-execute creates a control point that must be designed, tested, and monitored as part of the governance system — not added as an afterthought when an incident has already occurred.
Connect Approval Traces to Audit Evidence
An approval decision is only useful if it can be verified against what actually executed. A common gap in production governance is the disconnect between the authorisation layer and the execution record: the approval system shows that someone said yes, but nothing confirms that the executed action matched the approved payload.
Closing that gap requires linking two operational controls that are often built independently. The approval workflow captures the proposed action, reviewer identity, decision, and timestamp. The observability layer captures the actual tool calls, model outputs, and execution outcomes. Governance is the bridge between them.
When these layers are connected, a trace can answer: what action was proposed, who approved it, what payload was executed, did the executed payload match the approved one, what was the outcome, and what evidence was produced. This is where the trace contracts described in an AI agent observability runbook become the audit evidence layer. Every span should carry an approval ID or policy decision ID so that a reviewer or auditor can reconstruct the full proposal-to-execution chain.
This connection also surfaces control failures. A mismatch between approved and executed payload, an approval that fired on the wrong action, or a trace that lacks a matching decision ID are all governance incidents, not just observability gaps.
Build SLOs That Enforce Governance, Not Just Availability
Most service-level objectives for AI agents focus on uptime, latency, and throughput. Governance SLOs measure something different: whether the control system is working.
A governance SLO might target a minimum percentage of consequential actions that received an authorised decision before execution. It might measure the rate of payload mismatches between approved and executed actions, or the time between an action boundary violation and a containment response. It might track the fraction of high-risk agent runs that produced a complete, reviewable audit trail.
The NIST Generative AI Profile recommends ongoing monitoring and defined human-oversight responsibilities; it does not prescribe universal targets. The team must define its own indicators based on the consequence of each workflow. A good starting set includes:
- Approval coverage rate for actions in the "approve before execute" tier
- Approval-to-execution payload match rate
- Audit trail completeness per high-consequence workflow
- Cost-per-workflow anomaly detection coverage
- Time to detect and contain an unauthorised action
These indicators turn governance from a periodic review into a continuous signal. When a governance SLO is breached, the response is not a meeting — it is an investigation, a containment action, and a change to the control configuration.
Make Governance Readiness a Deployment Gate
Governance should not be reviewed after an agent is in production. It should be a pre-requisite that the team demonstrates before a workflow can run autonomously.
Practically, this means adding governance criteria to the deployment checklist that every agent release must pass. The criteria should include:
- Action classification documented and approved for each tool and workflow
- Approval routing configured and tested for every gated action
- Trace contracts emitting the fields needed for audit reconstruction
- Governance SLOs defined, measured, and meeting minimum thresholds
- Ownership assigned for each failure layer (model, retrieval, tool, policy, business outcome)
- Cost baseline established with anomaly alerting configured
- Rollback plan tested for model, prompts, policies, and index dependencies
This approach aligns governance with the deployment lifecycle instead of treating it as a separate process. Our enterprise AI pilot-to-production roadmap describes how readiness gates fit into a staged rollout — governance is one of the essential gates that a workflow must pass before moving from pilot to production.
The EU AI Act's human-oversight provisions in Article 14 provide a regulatory reference for deployment-time governance in covered high-risk systems. Legal applicability depends on jurisdiction and use case. The wider operational lesson is that teams should test whether oversight works before an incident, rather than assume a policy will be enough during one.
Assign Ownership, Escalation, and Cost Visibility
A governance framework that does not name owners is a governance framework that nobody operates. Each element of the control system must have an accountable team or role.
The failure-layer model from the observability runbook provides a useful starting point for ownership. Engineering typically owns orchestration and technical execution. Operations owns business-outcome monitoring and exception handling. Knowledge owners maintain retrieval quality and source freshness. Security owns identity, authorisation enforcement, and access reviews. Product owns workflow design and risk-tier decisions.
Each owner should know what signals they watch, what actions they can take, and what escalation path exists when they cannot resolve an issue alone. Escalation rules must be explicit: a policy ambiguity can escalate to the business system owner, a sustained cost anomaly can escalate to finance and engineering at a threshold defined from the measured baseline, and a blocked or bypassed authorisation gate can escalate to security.
Cost governance deserves particular attention because it is the governance dimension that most operational teams encounter first. A runaway agent loop is expensive, and cost anomalies are often the earliest signal that a governance control has failed — an agent that should have stopped at a boundary kept retrying, or a tool that should have been disabled kept producing results. Include cost-per-workflow baselines, anomaly thresholds, and automatic containment triggers such as rate limits, workflow-level spending caps, or automatic suspension when a budget is exhausted.
Microsoft's guidance on governance and security across the organization for AI agents describes how control plane, identity, lifecycle, and compliance layers interact in an enterprise deployment. The framework applies at any scale: start with one workflow, one owner, and one cost baseline, then extend as the agent portfolio grows.
Sources and Further Reading
- NIST AI Risk Management Framework — Authoritative risk management foundation for AI governance
- NIST Generative AI Profile — Voluntary guidance on monitoring and human oversight
- EU AI Act Article 14 — Human Oversight — Regulatory reference for human oversight provisions
- OWASP Excessive Agency (LLM06) — Security guidance on action boundaries and least privilege
- Microsoft: Governance and Security for AI Agents — Enterprise governance reference architecture
- OpenCraft: AI Agent Approval Workflow — Detailed guide to risk-based approval gates
- OpenCraft: AI Agent Observability Runbook — Trace contracts, SLOs, alert routing, and safe rollback
- OpenCraft: Enterprise AI Pilot-to-Production Roadmap — Staged rollout with governance readiness gates
Next Step
Governance is not built in a single sprint. Start with one production workflow, classify its actions, connect the approval trace to the observability record, and measure whether the control surface holds. Expand the framework as the agent portfolio grows.
If your team is designing governance for a live or planned agent deployment, an operational review can identify which controls are missing, which are ceremonial, and which are effective. Discuss an AI governance review with OpenCraft.