Finding categories
Every code block gets a status and a category. Only a few categories count as docs bugs. That's what keeps the signal clean.
Statuses
| Status | Meaning |
|---|---|
| pass | Executed (or syntax-checked) successfully. If the docs show output, it matched. |
| warn | Ran, but output differs from the docs, or a dependency emitted a deprecation warning. |
| fail | Exited with an error. The category says whether that's the docs' fault. |
| skip | Deliberately not run, with a stated reason. |
Docs bugs
A reader who copies the example as written will hit these. They appear in checks and get fix PRs.
| Category | Trigger | Real example |
|---|---|---|
api-drift | An import, attribute or keyword argument the installed version no longer has | Project A extensions.md:156; Project B llamaindex-react.mdx:78 |
invalid-syntax | The snippet doesn't parse, and isn't an intentional fragment | Project A transports.md:180 (missing comma) |
stale-output | The code runs, but the documented output is from an older runtime | Project A quickstart.md:118 (Python 2 u'' strings) |
package-not-found | The documented pip/npm package name doesn't exist | — |
Needs review
runtime-error, timeout and some network-external results are surfaced in the report but not filed as docs bugs automatically.
Not docs bugs
| Category | Meaning |
|---|---|
depends-on-context | Uses a name defined elsewhere on the page or elided by the author |
needs-credentials | Requires an API key or login |
network-external | A third-party service was unreachable or returned an error |
missing-install-step | Imports a package the page never installs (reported as a hint) |
missing-fixture | Expects a local file the reader is assumed to have |
missing-tool / host-tool | Needs a command-line tool or host resource outside the sandbox |
output-differs | Output varies by nature (timestamps, headers, ordering) |
deprecated-usage | A dependency printed a deprecation warning |
illustrative | Uses placeholder package or module names on purpose |
runner-gap / unsupported-syntax | A limitation of our runner (for example TypeScript-only syntax). Not reported as a docs bug |
Skipped on purpose
output-block (shown output, used for comparison), alt-variant (other tabs in a code group), fragment (contains ...), not-executable (config or prose blocks), unsupported-lang (languages we don't execute yet).
Measuring precision
We keep a hand-labeled gold set of 40 findings from three open-source projects, 7 of which are real docs bugs. Each engine change is scored against it. The first triage version reached 46.7% precision (8 false alarms). The current version scores 100.0% precision and 100.0% recall on the same set. With only 40 findings, treat these numbers as a regression test, not a guarantee of accuracy on your docs.