AI agent governance: the controls you need before deployment
The pressure to deploy AI agents is building across every enterprise. Vendors promise autonomous systems that handle customer enquiries, process transactions, and manage workflows without human intervention. The technology is genuinely impressive. The governance frameworks to manage it are almost universally missing.
I've reviewed AI implementations where agents had write access to production databases, could send external communications, and made financial decisions, all without logging, approval workflows, or rollback capabilities. These weren't reckless organisations. They were sophisticated enterprises that moved faster on deployment than on governance.
This gap creates real risk. Not theoretical risk that appears in consultant presentations, but operational risk that manifests as incorrect transactions, compliance violations, and reputational damage.
Why traditional governance doesn't fit
Most enterprise governance frameworks assume deterministic systems. Change management processes expect you to define exactly what a system will do before deployment. Audit controls assume you can trace from outcome back to rule. Compliance frameworks require documentation of decision logic.
AI agents break these assumptions. An agent's behaviour emerges from model weights, prompt engineering, and runtime context. You can't fully specify what an agent will do because its responses depend on inputs you haven't seen yet. The same agent might handle a customer complaint differently on Tuesday than Monday based on subtle variations in phrasing.
This doesn't mean agents can't be governed. It means governance approaches need to change. Instead of documenting what the system will do, you document the boundaries of what it can do. Instead of pre-approving specific actions, you implement runtime controls that catch problematic behaviour as it happens.
The five governance controls that matter
After reviewing dozens of agent implementations, I've identified five governance controls that separate manageable deployments from accidents waiting to happen.
Control 1: Action boundaries
Every agent needs technically enforced limits on what actions it can take. These aren't policy documents that describe intended behaviour. They're code-level constraints that prevent behaviour outside defined boundaries.
An agent handling customer service should have explicit permissions: it can retrieve order information, initiate refunds up to a defined threshold, and escalate to humans. It cannot modify account credentials, access other customers' data, or commit to contractual terms.
These boundaries must be enforced in the integration layer, not in the agent's prompt. The same rate limiting and authentication patterns that protect APIs from upstream floods should constrain agent actions.
Control 2: Decision logging
Every agent decision needs to be logged with sufficient context to understand why that decision was made. This means capturing not just the action taken, but the input that triggered it, the context the agent had access to, and any intermediate reasoning.
This logging serves multiple purposes. It enables post-incident analysis when agents behave unexpectedly. It provides audit evidence for compliance reviews. It creates training data to improve agent behaviour over time.
The logging architecture matters. Logs need to be immutable, timestamped, and stored separately from systems the agent can modify. If an agent can edit its own logs, you've created an unauditable system.
Control 3: Human escalation paths
Every agent needs defined conditions that trigger human review. These aren't just error conditions. They're uncertainty thresholds that route decisions to humans when the agent's confidence is low or when the stakes are high.
The escalation design requires careful thought. Too many escalations defeat the purpose of automation. Too few create uncontrolled risk. The goal is routing the right decisions to humans while letting agents handle routine cases autonomously.
Escalation triggers should include: decisions above defined thresholds, novel situations the agent hasn't encountered before, requests that touch sensitive data or systems, and any case where the agent's confidence falls below acceptable levels.
Control 4: Rollback capabilities
When an agent makes a mistake, you need the ability to reverse its actions. This sounds obvious, but many agent implementations lack transactional integrity that would enable rollback.
If an agent sends an email, can you unsend it? If it updates a database record, is the previous value preserved? If it initiates a payment, can you cancel it before settlement?
Rollback capability requires architectural decisions made before deployment. Systems integrated with agents need to support reversibility. Actions need to be tracked in ways that enable selective reversal. Recovery procedures need to be tested before they're needed.
Control 5: Drift monitoring
Agent behaviour changes over time. Model updates alter responses. Prompt modifications shift boundaries. Accumulated context affects decision patterns. Without active monitoring, agents drift from intended behaviour without anyone noticing.
Drift monitoring requires baseline metrics that define expected behaviour and alerting when actual behaviour deviates. What percentage of decisions should escalate to humans? What's the expected distribution of action types? How long should decisions take?
When metrics deviate from baselines, investigation should be automatic. Drift doesn't always indicate problems, but it always indicates change that needs to be understood.
Governance architecture in practice
These five controls don't exist in isolation. They form a governance architecture that surrounds the agent and mediates its interactions with other systems.
The PRISM methodology positions this governance layer in the Interface zone, where agents connect to the systems they interact with. Architecture assessments should evaluate whether this governance layer exists and whether it's comprehensive enough for the agent's scope of action.
The governance layer intercepts every agent action. It checks whether the action falls within defined boundaries. It logs the decision with appropriate context. It routes high-stakes decisions to human reviewers. It maintains the transaction history needed for rollback. It feeds metrics to drift monitoring systems.
This architecture adds latency and complexity. That's the cost of governed agents. The alternative, ungoverned agents with direct system access, is faster to deploy and catastrophically harder to manage when problems emerge.
The compliance dimension
Regulatory frameworks are catching up to AI agents, but they're not there yet. GDPR requires explainability for automated decisions affecting individuals. Financial regulations demand audit trails for transaction processing. Industry-specific requirements add additional constraints.
Current compliance frameworks weren't written with AI agents in mind, but they still apply. An agent that makes lending decisions is subject to fair lending laws regardless of whether a human or model made the decision. An agent that processes personal data is subject to privacy regulations regardless of whether it understands what it's processing.
Building governance controls now positions you for regulatory requirements that will inevitably tighten. Organisations that wait for explicit AI regulations will find themselves retrofitting controls into production systems, which is far more expensive than building them in from the start.
Starting with governance, not after
The most common mistake I see is treating governance as a phase that comes after deployment. Teams build agents, demonstrate capability, get approval to go live, and then start thinking about controls.
This sequence is backwards. The questions you ask at the start of an architecture engagement should include governance requirements. What decisions will this agent make? What are the consequences of wrong decisions? What controls are needed to keep risk acceptable?
Governance requirements should shape agent architecture, not constrain it after the fact. If an agent needs rollback capability, the systems it integrates with need to support that. If decisions need logging, the data architecture needs to accommodate log storage and analysis. If human escalation is required, workflow systems need to be in place.
Starting with governance doesn't slow deployment. It prevents the rework that comes from discovering governance gaps after agents are in production.
The governance maturity path
Not every organisation needs the same level of agent governance. The appropriate controls depend on what agents are doing and what happens when they make mistakes.
Low-stakes agents that recommend content or summarise documents need basic logging and monitoring. The consequences of mistakes are limited, so governance can be lighter.
Medium-stakes agents that interact with customers or process transactions need comprehensive controls. Action boundaries, decision logging, and human escalation become essential. Rollback capability becomes important.
High-stakes agents that make financial decisions or access sensitive systems need everything above plus enhanced audit capabilities, real-time monitoring, and formal approval processes for any changes to agent behaviour.
Assess your agents against this maturity model. If governance controls don't match the stakes, either reduce agent scope or increase governance investment.
If you're deploying AI agents without comprehensive governance frameworks, the PRISM AI Risk Audit identifies exactly where your controls fall short. $1,000 AUD.