AI Runbooks: How to Run AI Agents in Production
An AI runbook is the operating document that tells a named person how an AI system should behave, how to supervise it, when to stop it and how to recover it. The term is used in two ways, and it helps to separate them. Traditional runbooks are scripted procedures for IT tasks. AI agent runbooks govern a system that makes judgements, which is a different problem. This article covers both, then sets out what a production-grade AI runbook contains for a Malaysian bank, insurer, manufacturer or logistics firm.
TL;DR
A classic runbook (for example in Azure Automation) executes a script. An AI agent runbook defines the task boundary, permitted inputs, allowed actions, stop conditions and escalation path for a system that can fail plausibly.
Every AI runbook needs an owner, a definition of “normal”, a human stop rule, an evidence trail and a manual fallback.
In Malaysia, regulators already expect existing governance to be applied to AI, and a national AI Governance Bill is in consultation. A runbook is where that governance becomes daily practice.
Human correction of AI outputs is what separates a prototype from a production system, so build the feedback loop into the runbook.
Table of Contents
What an AI runbook actually is
One published definition describes an AI agent runbook as the operator-facing document explaining how a specific agent should behave, how to supervise it, what to do when inputs look odd, how to pause it safely and how to recover when something breaks. It is not an architecture diagram, a README, a prompt file or a post-incident review.
The practical test is whether an on-call operator at 2 a.m. can answer a handful of questions from the page alone. What wakes this agent up? Which systems can it read, and which can it write to? What does a normal day look like? Which failures are safe to retry, and which need a human to stop the process? Who approves risky actions? How do we fall back to manual work?
If the document cannot answer those, it is documentation of intent, not a runbook.
Script runbooks and AI agent runbooks are not the same thing
Search results for this term mix two ideas, so it is worth being precise.
In infrastructure teams, a runbook is usually a script. Azure Automation supports PowerShell and Python runbooks, and as of May 2026 the documented versions include PowerShell 7.6, 7.4 and 5.1, and Python 3.10. Runbooks can be started from the portal, through PowerShell cmdlets, by webhook or on a schedule. Tasks against local on-premises infrastructure need a Hybrid Runbook Worker.
Microsoft’s execution guidance adds the operational lessons: treat runbooks as transient processes that must tolerate restarts, and remember that the cloud sandbox has limits and does not support interactive operations. Job logs in Azure are kept for a maximum of 30 days, which matters if your audit obligations run longer. Export what you need to keep.
That is deterministic automation. Given the same input, the script does the same thing, and when it fails it usually fails loudly.
An AI agent is different. It interprets documents, drafts replies or recommends decisions, and it can produce a confident, plausible and wrong result without raising any error. The AI runbook therefore has to cover judgement: what the agent may decide, how its output is checked, and when a person takes over. The two kinds of runbook work together. The script runbook restarts a failed job. The AI runbook decides whether the job’s output can be trusted.
The sections every AI runbook needs
The structure below is the one we recommend to enterprise teams. Each section should fit on a page or two, in plain language that an operator who did not build the system can follow.
Purpose, owner and task boundary
Name the business process, the accountable owner and a named deputy. State what the agent is allowed to do and, just as clearly, what it is not allowed to do. An invoice-extraction agent that reads PDFs and proposes field values is a different risk from one that posts payments. Write that difference down.
Inputs, permissions and dependencies
List the document types, channels and data sources the agent may accept, and the systems it can read from and write to. Record the model or service versions in use and the upstream systems that, if they change, can silently alter behaviour. A new supplier invoice template or a revised policy wording is an input change, even if no code changed.
What normal looks like
Define the baseline numbers: volumes per day, accuracy on a sampled set, share of items routed to human review, average processing time. Without a baseline, drift looks like a quiet week. Set thresholds that trigger attention and thresholds that trigger a stop.
Operating modes and stop conditions
Most production agents need at least three modes: full operation, assisted mode in which every output is reviewed, and paused. The runbook should say who can switch modes, on what evidence, and how to do it. Stop conditions should be specific: accuracy on the daily sample falls below the agreed floor, review queues exceed a set size, a downstream system returns repeated errors, or a regulatory incident is suspected.
Exceptions and human review
Specify where uncertain or rejected items go, who clears them, within what time, and how their corrections are fed back. This section deserves the most care, as the next part explains.
Evidence capture and audit trail
For each decision, retain the input, the output, the model version, the confidence, who reviewed it and what they changed. In regulated work this is what lets you answer an auditor, a customer complaint or a regulator months later.
Incident response and manual fallback
Describe how to degrade gracefully. Which team handles the work by hand? Where are the manual procedures kept? How is the backlog reconciled when the agent returns? Test the fallback at least once before go-live, not for the first time during an incident.
Change control and retirement
A prompt edit, a model upgrade or a new document type is a change. The runbook should say who approves it, what regression tests run, and how to roll back.
Why the human feedback loop belongs in the runbook
The most useful lesson from document automation in banking is not about models at all. In a report on trade finance automation at a major international bank, more than 36 million trade documents are processed each year, and the approach relies on staff correcting the system’s classifications so that it improves over time. A system might start at around 60% accuracy with the goal of reaching 95%. Practitioners describe that human correction as the biggest difference between a prototype and a production-ready solution.
We agree, and it has a direct consequence for how you write the runbook. If corrections are an informal habit, they will stop when the team gets busy. If they are a defined step with an owner, a service level and a measure, accuracy keeps improving and the cost of review falls.
Results can be strong when this is done properly. In one 2026 case in the insurance sector, an insurer working with IBM and an integrator raised daily financial files processed by 750% at 97.6% accuracy, with automated reconciliation running ten times faster than manual work. That case comes from Brazil, not Malaysia, and your own figures will depend on document quality and process design. Treat it as evidence of what is possible, not a forecast.
The Malaysian context: BNM, PDPA and the AI Governance Bill
For Malaysian enterprises, an AI runbook is also a compliance instrument.
Bank Negara Malaysia’s discussion paper reports that 71% of banking institutions and DFIs had implemented at least one AI application in 2024, up from 56% the year before. The same paper finds that 92% of leading financial adopters apply existing governance and controls to AI, while 83% have set up new AI-specific oversight functions. The message for operations teams is that most AI risks are exacerbations of existing technology risks, so existing frameworks are often fit for purpose once recalibrated for the AI lifecycle. A runbook is exactly that recalibration: your incident, change and audit procedures, rewritten for a system that can be wrong without crashing.
On personal data, the Personal Data Protection Department’s guideline on automated decision-making and profiling notes that the Personal Data Protection Act 2010 has no specific provisions on automated decision-making, though processing of personal data that involves it still falls under the Act. If your agent touches customer data, the runbook should record the lawful basis, retention period and who may override an automated outcome. Confirm the details with your data protection officer or legal counsel.
Looking ahead, the National AI Office released a Public Consultation Paper on 10 July 2026 for a proposed AI Governance Bill covering developers and deployers across the AI lifecycle. The Bill is still evolving, so avoid hard-coding assumptions about its final obligations. Do build the habits it is likely to reward: a clear owner, documented behaviour, traceable decisions and a tested way to stop. HKTDC’s overview also links the Bill to the AI Technology Action Plan 2026-2030.
For a deeper treatment of standards mapping, see our guide to AI data governance under ISO/IEC 42001 and the Malaysian PDP framework.
A worked example: insurance claims document intake
To make this concrete, consider a general insurer that wants an agent to read incoming claim documents, classify them and extract key fields for the claims system. This is an illustration of how the sections apply, not a description of a specific client engagement.
The task boundary says the agent classifies and extracts but never approves, rejects or pays a claim. Inputs are scanned forms, emailed PDFs and photos from three channels. The baseline is set from a two-week pilot: the daily volume, the percentage of fields correct on a 5% sample, and the share of items sent to review.
The stop conditions say that if sample accuracy falls below the agreed floor on two consecutive days, the agent moves to assisted mode and every item is reviewed. If the claims system is unavailable, items queue and nothing is posted twice. Reviewers correct fields in the same screen they use for daily work, and those corrections are exported weekly for evaluation. The evidence record keeps the document, the extracted values, the model version and the reviewer’s change for the period the insurer’s retention policy requires.
Nothing in this is exotic. The value is that every question an operator might ask has an answer written before the incident, not during it. For document-heavy finance workflows, our article on intelligent document processing and hyper-automation in finance goes further, and IDP compared with OCR explains why extraction alone is not enough.
Our perspective: the runbook is written by whoever builds the system
Most runbook failures we see are not caused by missing templates. They happen because the team that designed the system, the team that built it and the team that operates it are three different groups, and the knowledge that matters is lost at each handoff. The runbook gets written at the end, by someone who was not in the room when the decisions were made.
Our approach is to have one senior team own strategy, build and operation, so the runbook is drafted during design and tested during build. The stop conditions are chosen when the use case is prioritised, because that is when the cost of an error is best understood. Governance, auditability and measured business results are engineering requirements from the first week, not a final review.
We also push clients to treat the runbook as a living asset. Monitoring and evaluation feed it, ongoing optimisation updates it, and an agent that cannot be explained to an operator is not ready to be promoted from pilot to production. For the pilot-to-scale path in operations, see our playbook on AI agents for operations.
— Thomas Samuel
Who owns the runbook, and how to roll it out
Assign ownership by process, not by technology. The business owner of the claims, trade finance or procurement process is accountable for the agent’s outcomes. Engineering owns the platform. Risk and compliance review the stop conditions and evidence requirements. Operations runs the daily checks.
Introduce the runbook in stages. Start with the agent in assisted mode so reviewers see every output. Run a tabletop exercise in which the team practises a stop, a fallback and a recovery. Only then widen the agent’s autonomy, and do so one measured step at a time. People adapt to new ways of working more readily when they have rehearsed them, which is why we pair this with AI change management for the teams affected.
Over the longer term, the runbooks for individual agents should sit within one AI operating model, so that roles, review cycles and escalation paths are consistent across use cases rather than reinvented for each.
Where Sentient Concepts can help
If you are moving an AI pilot into production, or you operate agents today without a tested runbook, we can help. We cover the full path from AI strategy and roadmap and readiness work, through AI and GenAI solutions engineering, to deployment and MLOps and managed operations. The same senior team stays accountable throughout. You can see the full range on our services page, or contact us to discuss a specific process.
FAQ
What is the difference between a runbook and a playbook?
In most operations teams a runbook is the step-by-step procedure for a specific system or task, while a playbook covers a wider scenario, such as a major incident, and may refer to several runbooks. Usage varies between organisations, so agree on the terms internally and apply them consistently.
Can an AI agent write its own runbook?
An AI tool can draft a runbook from incident notes or terminal sessions, which saves time. A named human who understands the process must still review and approve it, particularly the stop conditions, permissions and escalation rules. Treat the draft as a starting point and test it before relying on it.
How long should Azure runbook logs be kept?
Azure Automation keeps job logs for a maximum of 30 days, according to Microsoft’s documentation. If your audit, regulatory or internal retention rules require longer, export the logs to storage you control. Check your retention requirements with your compliance team.
Do we need an AI runbook for every agent?
Yes, for any agent that touches production data or customer outcomes. The sections can share a template, but the task boundary, thresholds and owners differ for each agent. A low-risk internal summariser may need a short page, while a claims or credit workflow needs a fuller one.
Sources
AI Agent Runbooks: The Missing Layer Between Demo Success and Production Revenue
AI and machine learning help Standard Chartered automate trade finance processing, Techgoondu
IBM e CXP implementam soluções de IA na MDS Brasil, IBM Brasil newsroom
Artificial Intelligence in the Financial Sector, Bank Negara Malaysia
Automated Decision-Making and Profiling Guideline, PDP Malaysia
Malaysia: Public Consultation on the AI Governance Bill, Baker McKenzie
MALAYSIA: New Legal Framework for Artificial Intelligence, HKTDC Research
Recommended