A good runbook template gives every on-call engineer the same thing at 3 a.m.: what the alert means, how to confirm it, the exact commands to fix it, how to check the fix worked, and who to call when it did not. This guide gives you a copy-ready runbook template in Markdown, explains what goes in each section, and walks through a complete filled-in example for a real alert: a log disk filling up.
About the example: we ran every command in it on 18 November 2026 on macOS 27.2, against a test directory filled with 30 dummy compressed logs, with the path swapped in. The commands also use flags that work with GNU tools on Linux, but we did not run them on a Linux server. The service, team and incident names are invented for the example.
TL;DR
- Write one runbook per alert or recurring task, named after the alert exactly as it appears in the page.
- Use a fixed structure: summary, impact, before you start, confirm, find the cause, fix, verify, escalate, rollback.
- Every command must be copy-paste ready and tested. Mark destructive steps and say how to undo them.
- Link the runbook from the alert itself, and record when it was last tested and reviewed.
What Is a Runbook?
A runbook is a written, step-by-step procedure for a specific operational task or alert: restarting a stuck worker, rotating a certificate, freeing disk space, failing over a database. Its reader is under time pressure and may not know the system well, so it trades explanation for clear, ordered actions. Google’s SRE teams call the same document a playbook; Google’s Site Reliability Engineering book reports that recording best practices ahead of time in a playbook gave them roughly a 3x improvement in mean time to repair compared with “winging it”.
The terms overlap and teams use them loosely. A useful way to separate them:
| Document | Scope | Example |
|---|---|---|
| Runbook | One task or alert, with exact steps and commands | “CheckoutApiLogDiskHigh: free space on the log volume” |
| Playbook | A wider response process: roles, communication, decisions; may link to several runbooks | “How we run a SEV-1 incident” |
| SOP (standard operating procedure) | A routine, planned procedure, often for compliance | “Monthly access review” |
For the incident process around runbooks (roles, severity levels, status updates), PagerDuty’s open incident response guide is a well-known public reference.
Copy-Ready Runbook Template (Markdown)
Copy this into a .md file next to your service code or into your wiki. Replace everything in angle brackets.
# Runbook: <alert or task name>
| Field | Value |
|----------------|-----------------------------------------------|
| Service | <service name> |
| Owner | <team> (<chat channel>) |
| Alert | <exact alert name, as it appears in the page> |
| Severity | <SEV level and what it means for users> |
| Last tested | <YYYY-MM-DD> by <name> |
| Last reviewed | <YYYY-MM-DD> |
## Summary
One or two sentences: what is wrong when this alert fires and
what this runbook gets you to.
## Impact
What users notice. What breaks next if nobody acts, and how soon.
## Before you start
- Access you need (hosts, dashboards, roles)
- Tools you need, with versions if it matters
- Who to tell that you are working on it
## 1. Confirm the problem
Commands or dashboard links that prove the alert is real.
Show what "bad" and "normal" look like.
## 2. Find the cause
Ordered checks, most likely cause first.
Each check: command, what to look for, which step to go to next.
## 3. Fix
Numbered steps. Copy-paste commands. Mark anything destructive
with WARNING and say how to undo it.
## 4. Verify
How to prove the fix worked: the same check as step 1, now normal.
How long to watch before closing.
## Escalate if
- <condition>, then page <team or person> via <tool>
- Fix did not work within <N> minutes
## Rollback
How to undo step 3 if it made things worse.
## Related
- Dashboard: <link>
- Alert definition: <link>
- Past incidents: <links>
## Change log
- <YYYY-MM-DD> <name>: <what changed>What Goes in Each Section
Header table
The alert name must match the page character for character, so the on-call engineer can search for it. Last tested is the most important field: a runbook nobody has run in a year is a guess. Keep the owner as a team and channel, not a person who may have left.
Summary and impact
The summary answers “what is happening and where will this runbook get me?” in two sentences. The impact section answers “how urgent is this?”: what users see now, what breaks next, and roughly how long until it does. That time estimate is what lets someone decide between fixing it now and waiting for daylight.
Before you start
List the access and tools needed up front. Finding out at step 3 that you lack a production role wastes the minutes that matter. Include the “tell the channel” step so two people do not work on the same alert.
Confirm, find the cause, fix
Keep these three apart. Confirm proves the alert is real and shows what normal looks like. Find the cause is a short decision tree: each check says what to look for and which step to go to next, most likely cause first. Fix is numbered steps with commands that can be pasted as they are. Put a dry run (list or count what will change) before every destructive command.
Verify, escalate, rollback
Verify repeats the confirm check and says what result counts as fixed and how long to watch. Escalate if gives concrete conditions, not “if unsure”. Rollback says how to undo the fix, or states plainly that it cannot be undone and why that is acceptable.
Related links and change log
Link the dashboard, the alert definition and past incidents. The change log shows readers whether the runbook is maintained; the same habit keeps release notes trustworthy.
Runbook Example: A Log Disk Filling Up
Here is the template filled in for a common alert: the disk holding an API’s logs passes 90%. It is the kind of runbook a team would keep in the service repository.
# Runbook: checkout-api log disk above 90%
| Field | Value |
|----------------|---------------------------------------------------|
| Service | checkout-api |
| Owner | Payments platform team (#payments-oncall) |
| Alert | CheckoutApiLogDiskHigh |
| Severity | SEV-3; SEV-2 if usage passes 97% |
| Last tested | 2026-11-18 on a test directory |
| Last reviewed | 2026-11-18 |
## Summary
The volume that holds /var/log/checkout-api is more than 90% full.
This runbook frees space by removing compressed logs older than
14 days, which are already shipped to central logging.
## Impact
None yet. At 100% the service cannot write logs and health checks
start failing, which takes instances out of the load balancer.
At normal log volume that is about 6 hours after the alert.
## Before you start
- SSH access to the checkout-api hosts (role: payments-oncall)
- Post "Looking at CheckoutApiLogDiskHigh on <host>" in #payments-oncall
## 1. Confirm the problem
df -h /var/log/checkout-api
Look at the Capacity (macOS) or Use% (Linux) column.
Normal is under 70%.
## 2. Find the cause
Biggest directories first:
du -xh -d 1 /var/log/checkout-api | sort -rh | head -n 5
- archive/ is the largest: old rotated logs. Go to step 3.
- current/ is the largest: the app is logging too much.
Do not delete current logs. Escalate (see below).
Check for deleted files that are still open (space not released):
lsof +L1 | head -n 5
- Any checkout-api process listed: restart it with the deploy
tool's restart action, then go to step 4.
## 3. Fix
Count and size the files that will be removed:
find /var/log/checkout-api -type f -name "*.gz" -mtime +14 | wc -l
find /var/log/checkout-api -type f -name "*.gz" -mtime +14 -exec du -ch {} + | tail -n 1
WARNING: the next command deletes files and cannot be undone.
Run it only after the two commands above show what you expect.
find /var/log/checkout-api -type f -name "*.gz" -mtime +14 -delete
## 4. Verify
df -h /var/log/checkout-api
Usage should be under 70%. Watch the alert for 15 minutes;
it clears on its own once usage drops below 85%.
## Escalate if
- current/ is the largest directory: page the checkout-api
developer on call (log level or a retry loop is the usual cause)
- Usage is still above 90% after step 3
- Usage passes 97% at any point: raise to SEV-2
## Rollback
Deleted archives cannot be restored on the host. Copies older than
14 days are in central logging, so nothing is lost.
## Related
- Dashboard: checkout-api host disk
- Past incidents: INC-2041 (debug logging left on after a release)
## Change log
- 2026-11-18: first versionWhat our test run showed
We created a test directory with 30 compressed logs of 3 MB each, dated 1 to 30 October 2026, plus a 5 MB current log, and ran the example’s commands against it:
du -xh -d 1 … | sort -rh | head -n 5listedarchive/at 86M andcurrent/at 4.8M, so the decision tree pointed to step 3.- The count command returned 30 and the size command returned
86M total, matching what we created. find … -mtime +14 -deleteexited with status 0, removed all 30 archives and leftcurrent/app.loguntouched;du -shthen showed 4.8M.lsof +L1ran and listed open files with a link count of zero (deleted while still open). On our Mac these were system processes, not an app we could restart, so the restart branch was not tested.
Two details worth copying into your own runbooks. First, -d 1 sets the depth for du on both macOS and GNU/Linux, while --max-depth=1 is GNU-only. Second, -mtime +14 matches files modified more than 14 full days ago; see the find manual page for the exact rounding rules before you rely on the cut-off.
How to Write a Runbook, Step by Step
- Start from your alerts. List the alerts that paged someone in the last quarter. Each one that needed a human gets a runbook, starting with the most frequent.
- Write it while doing the task. The next time the alert fires or the task is due, record each command and its output as you go. Notes written from memory skip the steps that seem obvious to the expert.
- Fill the template. Put the commands into the confirm, cause and fix sections. Replace hostnames with variables or clear placeholders only where they really vary.
- Add the safety rails. A dry run before every destructive command, a WARNING line, and a rollback section.
- Test it with someone else. Ask an engineer who did not write it to follow it on staging or a test machine, exactly as written. Every question they ask is a missing line.
- Link it from the alert. Most alerting tools let you attach a runbook URL to the alert so it appears in the page. A runbook that has to be searched for is often not found.
- Review it after every use. If a step was wrong or missing, fix it before closing the incident and add a change log line.
The SRE Workbook chapter on on-call makes the same point about upkeep: create the playbook entry when you create the alert, keep it current, and replace long lists of deterministic commands with automation over time.
Common Runbook Mistakes
- Explaining the architecture instead of giving steps. Link to the design docs; keep the runbook to actions.
- Commands that only work on the author’s machine. Shell aliases, local paths and tool flags that differ between macOS and Linux all break at the worst time.
- No “normal” value. “Check the queue depth” is useless without saying what number is healthy.
- Vague escalation. “Escalate if needed” leaves the decision to the person least able to make it. Name conditions and a time limit.
- Never tested. Add the last-tested date to the header and treat an old date as a bug.
Where to Keep Runbooks
Keep runbooks where engineers already look and where changes are reviewed. Many teams store them as Markdown in the service repository, so a code change and the runbook change land in the same pull request; our guide to version control for documentation covers that workflow. A wiki works too, as long as each page has an owner. Either way, plan the structure so runbooks are easy to find by alert name; see building a documentation architecture. And when a runbook step depends on reading an error message, good error documentation shortens the find-the-cause step a lot.
Conclusion
A runbook is only as good as its last test. Start with the template above, fill it in the next time an alert fires, and have someone else run it before you trust it. Keep the steps short, the commands exact, the destructive actions guarded, and the escalation rules concrete, and the runbook will do its job when the person reading it is tired and under pressure.
What should a runbook template include?
At minimum: the alert or task name, owner, severity, a short summary, impact, prerequisites, steps to confirm the problem, find the cause and fix it, how to verify the fix, escalation conditions, rollback steps, related links, and the dates it was last tested and reviewed.
What is the difference between a runbook and a playbook?
A runbook covers one task or alert with exact steps. A playbook usually covers a wider process, such as how a team runs an incident, and can link to several runbooks. Google’s SRE teams use “playbook” for what many teams call a runbook, so check how your team uses the words.
How long should a runbook be?
As short as it can be while staying complete. Most single-alert runbooks fit on one or two screens. If it grows much longer, split it by cause or move background explanation to a linked design document.
Who should write runbooks?
The engineers who own the service and handle its alerts, ideally while doing the task. Someone else should then test it by following it exactly as written.
Where should runbooks be stored?
Where on-call engineers will find them fast: in the service repository as Markdown, or in a team wiki. Link each runbook from its alert so it appears in the page.
How often should you update a runbook?
After every use, and on a regular review, for example every quarter. Record each change in the change log and update the last-tested date when someone runs it end to end.







