On this page
What documentation drift isWhy code examples drift firstSignals that you already have driftHow to detect it, from cheapest to most thoroughA simple drift-prevention workflowMeasuring driftCatch it before your users doWhat documentation drift is
Documentation drift is the gap between what your docs say and what your product does. It comes in three kinds, and they fail differently:
- Reference drift. Parameters, endpoints, defaults and error codes in the API reference no longer match the code. Generated references drift less, but hand-written tables and descriptions around them still do.
- Conceptual drift. Guides that explain how something works, or recommend a pattern, after the product changed how it works or what it recommends. Nothing errors. Readers just build the wrong thing.
- Example drift. Code samples, commands and printed output that no longer run or no longer match. It's the most visible kind, because readers copy examples word for word and find out within seconds.
A fourth cause cuts across all three and is easy to miss: dependency drift. Your docs import other people's packages, and those packages ship releases on their own schedule.
Why code examples drift first
An example is the most specific claim in your docs. A conceptual page can stay roughly true for years. An example names exact imports, exact keyword arguments and exact output, so any change to any of them breaks it. Here's what that looked like in our own runs on open-source docs sites, as published on the results page:
- A keyword argument the API stopped accepting. On Project A, two examples on one page pass a keyword argument the function no longer takes in the current release.
- Classes a dependency stopped exporting. On Project B, a cookbook imports an MCP client class and another imports an agent class, each from a third-party framework. Both frameworks' current releases no longer export those classes. The docs didn't change. The packages under them did.
- Output from an older runtime. An example still runs, but the output printed under it is in a format current interpreters no longer produce.
In a terminal, the first two look like this (names redacted until we've reported them upstream):
TypeError: <function>() got an unexpected keyword argument '<param>'
ImportError: cannot import name '<RemovedClass>' from '<agent-lib>'
Neither error needs domain knowledge to find. Each one needs someone, or something, to run the page against today's versions.
Signals that you already have drift
You don't need a tool to know whether drift is costing you. Look for:
- Support tickets that quote your docs. "I followed the quickstart and got…" is drift reported by a customer.
- Issues labelled "docs" that mention an error message. Those are bugs in docs, not requests for more docs.
- Error strings in community channels. People paste the traceback from your example into forums, Discord or Stack Overflow before they file anything.
- Workarounds in comments. "This example is outdated, use X instead" under a tutorial, a gist or a video means readers are patching your docs for each other.
- Fixes that arrive from outside. Drive-by pull requests that change one import line are drift your own team didn't catch.
How to detect it, from cheapest to most thorough
| Method | Catches | Misses | Cost |
|---|---|---|---|
| Docs owners on code reviews | Changes to public APIs that obviously need a docs update | Anything reviewers don't think of, and all dependency drift | Minutes to set up |
| Release checklist | Forgotten migration notes and changelog entries | Examples nobody re-ran | A template and some discipline |
| Link checking | Moved or deleted pages and anchors | Content that's wrong but still loads | Low, runs in CI |
| OpenAPI diffing | Breaking changes between spec versions | Hand-written examples and guides | Low if you have a spec |
| Executing code samples | Examples that error or print different output, including dependency drift | Conceptual drift, and examples that need real credentials | Medium, unless a service does the triage |
Put docs owners on code reviews
GitHub's CODEOWNERS file requests reviews from the owners of the paths a pull request touches. Add your docs team to the code paths that define the public API, not just to docs/:
# .github/CODEOWNERS (the last matching pattern wins)
/docs/ @your-org/docs
/src/client/ @your-org/sdk @your-org/docs
/openapi.yaml @your-org/api @your-org/docs
It costs nothing and catches the drift someone notices. It can't catch what nobody notices, and it can't see third-party releases at all.
Add docs to the release checklist
A short block in your pull request or release template makes the question unavoidable: Did a public signature, default or output change? Is the change in the changelog and, if it breaks something, in the migration guide? Are deprecated names removed from examples, not just from code?
Check links on every pull request
A link checker such as lychee fails the build when a page or anchor disappears. It's cheap, and it catches restructures. It says nothing about whether the content behind a working link is still right.
Diff your OpenAPI spec between releases
If your reference is generated from an OpenAPI file, a diff tool such as oasdiff lists the changes between two versions of the spec, and its breaking command shows only the changes that break existing clients. Wire it into the release so every breaking change has a docs ticket. It covers the reference, not the examples around it.
Execute the code samples
The only check that finds example drift directly is running the examples: on each release against what you're about to ship, and nightly against the latest versions of your dependencies, because that's when third-party drift shows up. How to set that up, and how to keep the results readable, is covered in our guide to testing the code samples in your docs. If you ship an SDK or client library, the SDK docs release-gate guide covers the version gap between your docs and the package readers install.
A simple drift-prevention workflow
The order matters less than the loop. Code changes pull in a docs reviewer, releases re-run the examples against what's shipping, nightly runs catch dependencies moving, triage keeps the signal clean, and every fix goes through review like any other change.
Measuring drift
Measure what's left after triage, not raw pass rates. Raw numbers mislead: on Project B, 406 of 927 tested blocks passed, yet only 7 were flagged as likely docs bugs. Most failures were blocks that depend on earlier context or need credentials, which is normal for docs and not drift. Three numbers are worth tracking:
- Likely docs bugs open, after triage, per docs site or section.
- Time from flag to fix. Drift you find in a nightly run and fix the same week never reaches a ticket.
- Pages covered by an execution check, out of pages with runnable code. Drift hides in the pages nobody tests.
Be careful with confirmed-versus-flagged too. Of the 11 flags across our three runs, we've confirmed 6, found 2 false alarms (a deliberate "before" example in a migration guide, and a template placeholder) and 3 are still unreviewed. If you report a number, say which one it is.
Catch it before your users do
DocsAlive runs the code samples in your docs in an isolated container on each release and nightly, separates likely docs bugs from expected failures, and can draft the fix for your team to review. To see what drift your docs have today, request a free health check: our team runs it on a public repository and emails you the real report.
Comparing tools for drift? Promptless vs DocsAlive covers an AI agent that drafts docs updates from what you ship, and Doc Detective vs DocsAlive covers an open-source docs-as-tests framework.
