AI Agent Governance Framework

1. Start with governance principles

A governance model should begin with a small number of principles that apply consistently across the agent estate. These principles should be understandable by product owners, developers, risk functions, compliance teams and business users.

The most important principles are traceable ownership, least privilege, proportional control, human accountability, evidence by design and controlled change. Higher-risk agents should require stronger controls rather than forcing the same process on every use case.

• Every agent has a named business owner and a named technical owner.
• Every agent operates within an explicit purpose and authority boundary.
• Access to systems, data and actions follows least-privilege principles.
• Human oversight is designed according to the consequences of error.
• Testing and approval evidence is retained and reproducible.
• Material changes trigger reassessment rather than silently expanding capability.

2. Maintain an authoritative agent inventory

Organisations cannot govern agents they cannot identify. An enterprise inventory is therefore the foundational governance control.

The inventory should not be a simple list of names. It should provide enough information for governance teams to understand what the agent does, where it runs, which systems it can access and who is accountable for it.

• Agent name, version and lifecycle status.
• Business purpose and intended users.
• Business owner, technical owner and support owner.
• Microsoft environment, solution and deployment location.
• Connected systems, connectors, tools and data categories.
• Action permissions and transaction authority.
• Risk classification and required approval level.
• Human-oversight model.
• Test evidence, certification state and last review date.

3. Classify risk before deployment

Risk classification determines how much governance an agent requires. The important variables are not only the model being used, but what the complete agent can see, decide and do.

A read-only information assistant is fundamentally different from an agent that can approve transactions, communicate externally, modify records or trigger downstream processes.

The CAI Score is ColleagueAI's structured approach to classifying enterprise agent risk. It considers autonomy, impact, data sensitivity, tool access, reversibility, oversight and governance requirements. Classification should occur before deployment and again when material capability changes.

4. Separate business accountability from technical ownership

Technical teams can build and maintain an agent, but they should not become the sole owners of the business outcome. The business function using the agent must remain accountable for the process and consequences.

A mature model therefore distinguishes business ownership, technical ownership and governance assurance. This prevents accountability from disappearing between technology and operations.

The ownership model should also define decision rights. The business owner should approve the intended purpose, acceptable outcomes and material process changes, while the technical owner remains responsible for the integrity of the solution components, configuration and technical controls. Governance, risk, security or compliance functions can provide challenge and assurance without becoming substitutes for accountable business ownership.

5. Govern identity, access and tool permissions

Agent capability is largely determined by the systems and actions available to it. Governance must therefore treat connectors, credentials, service identities and action permissions as first-class controls.

Permissions should reflect the minimum required to fulfil the intended use case. Where possible, high-impact actions should be separated from low-risk retrieval and analysis functions.

• Use dedicated identities where the platform and architecture support them.
• Restrict connectors and actions to approved use cases.
• Avoid broad permissions merely for development convenience.
• Record privileged actions in auditable logs.
• Use approval gates for irreversible or high-impact actions.
• Revalidate permissions when the workflow or integration changes.

6. Design human oversight as part of the workflow

Human oversight should not be a generic statement that a human remains responsible. It should identify exactly where people review, approve, intervene or stop the agent.

The appropriate model depends on risk. Some agents can operate autonomously within tightly bounded low-risk tasks. Others require human approval before a consequential action. High-risk use cases may require continuous supervision or may not be suitable for autonomous execution at all.

The control should specify the point of intervention, the reviewer, the information available to that reviewer and the action the reviewer is authorised to take. A nominal approval step is not sufficient if the person cannot understand the proposed outcome or cannot stop it. Oversight should therefore be tested as an operational control rather than documented only as a policy statement.

7. Test the agent as a system, not only as a prompt

Agent assurance requires testing of the complete workflow. Prompt quality matters, but so do permissions, tool calls, failure behaviour, data handling, escalation logic, output quality and recovery from unexpected inputs.

A production package should preserve evidence of what was tested, the expected behaviour, known limitations and the version that passed the test.

• Functional and scenario testing.
• Permission and connector testing.
• Failure and exception paths.
• Human-approval and escalation paths.
• Sensitive-data handling.
• Output-quality criteria.
• Logging and evidence generation.
• Regression testing after material changes.

8. Make deployment a controlled handover

A packaged agent should arrive with a clear deployment boundary. The vendor can provide the tested solution package, documentation, configuration guidance, governance evidence and deployment instructions, but the customer controls its own tenant and operating environment.

For ColleagueAI products, deployment and runtime remain in the customer's Microsoft environment. The customer or its approved implementation partner performs tenant-specific configuration, integrations, deployment and operation.

A controlled handover should also identify prerequisites, environment variables, connection references, required roles, known limitations and post-import validation steps. That allows the customer or implementation partner to reproduce the tested configuration without relying on undocumented knowledge from the original builder. The deployed version should remain traceable to the package and evidence that were approved.

9. Monitor operating risk and material change

Approval at launch is not the end of governance. Agent behaviour can change when prompts, models, tools, permissions, knowledge sources or connected systems change.

Continuous governance should therefore focus on material changes, exceptions, incidents, usage patterns, approval overrides and evidence that the deployed configuration still matches the approved design.

Monitoring should be connected to explicit review triggers. Examples include a new connector, broader permissions, a different model, a new knowledge source, a rise in overrides, repeated exceptions or a change in the business process. Those signals should determine when the agent requires investigation, remediation, recertification or a fresh governance decision rather than remaining indefinitely approved.

10. Govern retirement as seriously as launch

Unused or superseded agents should not retain dormant access indefinitely. Retirement should remove permissions, disable integrations, archive evidence and update the enterprise inventory.

A clean retirement process reduces attack surface and prevents organisations from accumulating unowned AI capabilities.

Retirement should also close operational and governance dependencies. Owners should confirm that scheduled flows are disabled, service credentials are revoked where appropriate, downstream integrations no longer expect the agent, retained evidence meets policy requirements and users are directed to the replacement process if one exists. The inventory should record the retirement date and final disposition.

Frequently asked questions