Testing README code examples

The short answer

Test a README as the first sequence a reader follows: start in a clean environment, run the documented install command, execute the usage examples in order, and check the results the page promises. Test the version and installation source the reader will actually use. Record prerequisites, skipped paths and any setup supplied by the test, so a passing result does not conceal a missing step.

Run focused checks on pull requests, test a candidate artifact before publication when you need a release gate, and repeat them on a schedule to find dependency and installation drift. For the broader practice, start with how to test documentation code samples.

DocsAlive extracts supported fenced examples and runs or checks them in page order. Its reports classify findings and show skips; credentials are classified rather than used to execute examples, compiled-language examples are not executed yet, and TypeScript support is limited. Review the report alongside the actual coverage rather than treating a pass as proof of the whole README.

Why README examples break

A README often supplies the first install command, a short usage example and a link to deeper documentation. It may be the only page a prospective user reads before deciding whether the project works. That makes even a small break expensive: a renamed package, an import that moved, or an option removed from a command can stop the first successful run.

The README also sits at several boundaries. Repository instructions can describe the current development branch while a package registry serves a release from last month. A command may work for a maintainer because a dependency is already installed or a local configuration file supplies a value. An example can execute successfully but print different output from the result shown on the page.

Consider the promise behind a simple sequence: install a package, import it, call a function, and see a result. Each step depends on the last. A syntax check on the final block will not prove that the install command finds the intended distribution. A test that installs the project from the checkout may pass even when the package published to a registry is incomplete. A successful command can still produce stale output or rely on an undocumented environment variable.

Testing these promises is part of managing documentation drift. The goal is not to run every line indiscriminately. It is to know which reader paths were exercised, which assumptions the test supplied and which claims remain unverified.

Map the reader's path

Read the README from the beginning as a new user would. List the supported platforms, prerequisites, installation alternatives and first useful outcome. Identify which blocks form a sequence and which are alternatives. A Python and a JavaScript tab may illustrate the same task, but executing one does not establish that the other works.

Assign an owner to the important paths. Start with the install and first successful use; then add common configuration, supported installation alternatives and error recovery. Inventory examples that cannot be executed unattended rather than leaving them invisible. A small list with owners and explicit gaps is more useful than a large suite whose actual coverage no one understands.

The page may contain badges, generated content, screenshots and prose in addition to code. Execution is one layer of review. Use a link checker for destinations and a person for the explanations, visual presentation and whether the examples answer a real reader question. The free tools guide covers other layers.

Start with a clean environment

Create an environment that resembles the documented starting point. If the page says only that Python is required, begin with the stated supported Python version and an empty environment. Do not inherit the maintainer's project dependencies, shell aliases, credentials or configuration. Install any operating-system tools the README explicitly requires, and record them with the test.

For a Python project, the following illustrates the isolation step; substitute the runtime and commands appropriate to the project:

python -m venv /tmp/readme-check
. /tmp/readme-check/bin/activate
python -m pip --version

The environment alone is not the test. Follow it with the install command from the README and then the usage sequence. If the README asks a reader to clone the repository first, include that step. If it promises installation from a registry, installing a local checkout instead answers a different question. Keep any test harness outside the source tree when local files could accidentally shadow the installed package.

Record the operating system, runtime, package source, package version and network assumptions. Where a project supports several versions, choose a documented matrix and make its limits visible. A Linux check does not prove a Windows command works, and a check on the newest runtime does not establish compatibility with the oldest supported runtime. The right matrix depends on the paths the project promises, not on every combination a runner could create.

Run the install the reader sees

Copy the install command as documented. Verify the distribution name, version constraint, registry or repository URL, and any extras or feature flags. A successful installation should be followed by a check that the intended artifact is the one imported or invoked. An old global executable on the runner can otherwise make a broken install look successful.

A package name and import name do not always match. If the install command uses one name and the example imports another, the test should preserve that distinction rather than rewriting either side. Similarly, a command that depends on a separate system library may install the Python package successfully but fail at first use. The README should state that prerequisite if a reader must provide it.

When the install requires network access, distinguish an unavailable registry from a wrong package name or version. Retain the error and the target source. A retry can help diagnose a transient outage, but repeated retries should not silently turn a recurring installation problem into a pass.

Test checkout, candidate and published versions separately

These checks answer different questions:

For a release gate, build the candidate, install it in a fresh environment and run the examples before publication. Record the artifact and version. After publication, a separate check of the public install path can detect packaging or registry problems, but it cannot block a release already shipped. The SDK docs release-gate guide goes deeper on testing the built package.

A README in a repository may intentionally describe unreleased behavior. If so, say which version it targets and maintain a way for readers of an earlier release to find matching instructions. Otherwise a test can correctly pass on the default branch while users of the stable package encounter an unavailable method.

Execute the usage sequence

Preserve the order in which readers encounter dependent blocks. An import in the first block can be needed by a call in the next; a shell command can create the file consumed later. Start a new independent example from a clean state when the README presents it as independent. If the page relies on a file, dataset or earlier command, make that relationship clear.

A useful first-use example is complete enough to copy, with imports, a small input and a visible outcome. A fragment can be appropriate when it illustrates one detail, but label it as a fragment and explain the surrounding context. Avoid silently adding missing imports in the test harness: that may make the test pass while leaving the reader with an error.

For example, a README might show an illustrative calculation and result:

values = [2, 4, 6]
mean = sum(values) / len(values)
print(mean)

4.0

An exact output check is reasonable here because the input and result are stable. For a real package, replace the calculation with the documented invocation and use the actual artifact under test. If a later example changes a configuration file or working directory, decide whether that state is part of the documented sequence or a separate path.

Check more than the exit status when the README promises a result. An example can exit successfully while returning an empty list, writing to the wrong directory or using a default different from the page. Assert the meaningful behavior and explain the result a reader should expect. When the example intentionally changes state, inspect the controlled state as well as the process result.

Keep test setup visible

A harness may need to prepare an isolated directory, a small input file or a local fixture. Record what it supplies, and compare that list with the prerequisites on the page. If the harness supplies a configuration value that the reader is never told to set, the README may be incomplete even though the command passes.

Sometimes a fixture stands in for a remote service. That makes an example repeatable and can check how code handles a response, but it does not show that the live service is available or returns the same data. Label the target in the report. Use a separate authorized integration test when the live behavior is part of the promise you need to verify.

A setup command that installs the checkout can also hide a broken documented installation. Test the README install line independently or make that limitation clear. Do not insert a helpful environment variable, secret or executable into a test and then report the page as self-contained.

Compare output without hiding changes

Output can be a contract, an illustration or a partial excerpt. The page should tell readers which. A stable string or a required field can be compared exactly; a generated identifier, timestamp or unordered collection may need a narrower assertion.

Normalize only known variable fields. Removing every differing line can hide a renamed field or a missing error. Keep expected and actual evidence so a reviewer can decide whether the documentation, the implementation or the comparison should change. If the README shows a representative response, say which values are examples and which shape or fields the user should rely on.

Output also includes files, exit codes and side effects. A command that prints success but returns a failure status can break a script; a quiet command can succeed. Check the behavior the page describes rather than assuming that text on stdout is the only useful result.

Handle credentials, interaction and external state

Not every README example should run in the same unattended job. A command may require a secret, open a browser, wait for input, start a long-running server or modify a remote resource. Identify these cases and select an appropriate separate check. Do not put real keys in documentation, fixtures or logs.

A timeout is evidence to investigate, not an automatic verdict that the README is wrong. The process may be waiting for a prompt, an external service may be unavailable, or the command may be intended to remain running. Preserve the command and environment with the result. For credentialed behavior, use a separate authorized environment and a narrowly scoped test account when appropriate.

DocsAlive classifies examples needing credentials rather than executing them with real keys, including customer-provided test credentials. Interactive, graphical, destructive and long-running commands need a deliberate separate strategy; a skip or timeout does not establish whether the reader's full path works.

Put README checks in CI

Use different runs for different questions. A pull request check can catch a renamed import before merge. A pre-publication check can test the candidate artifact. A scheduled run can find a dependency or registry change even when the repository has not changed.

Keep a fast first-use test in the pull request path. Run broader platform and dependency matrices at a cadence that fits the project, and make the delay explicit. Include changes to the package and build configuration as triggers, not only edits to the README: code and packaging changes can break an unchanged page.

For a practical workflow, create a clean environment, install the chosen target, execute the documented sequence and retain the output and version. Make a failing stable assertion visible to the pull request author. Give uncertain network failures and unsupported paths an owner rather than suppressing them. Review the job periodically: a scheduled check that no longer runs gives no evidence of current health.

Use DocsAlive for supported examples

The CLI accepts a local directory or Git URL. Without --execute, it extracts and plans rather than running examples. For a focused local scan, adapt the include pattern to the repository and inspect the selected files:

docsalive scan . --execute --include "README.md" --out out

The pipeline extracts fenced blocks and an output block immediately following a block. It keeps blocks in page order and selects one variant from each tab group. Each page receives a fresh container. Python, JavaScript on Node 20, simple TypeScript, shell and curl or HTTP examples have the coverage described in the framework reference; compiled-language examples are recognized but not executed yet. Other tab variants require separate coverage.

The default environment includes Python 3.12 and Node 20. A project may need a different image or setup. If setup installs the project, record whether it installed the checkout or a candidate artifact and check that it has not hidden an omitted README instruction. The CLI reference describes the available options.

Read report.md and report.json with the findings reference. A pass may represent execution or a syntax check; warnings, failures and skips require context. The report identifies code blocks and reasons, but it cannot prove that an unselected alternative, a live credentialed flow or a missing example works.

Decide what a failure means

A useful triage policy separates the reader-facing problem from an environmental failure without discarding either. Record the page, line, command, tested artifact, runtime, setup, actual output and expected behavior. Give an owner a way to reproduce the result.

A test failure is not always a documentation bug, and a passing check is not always execution. Review status, category and reason together. When a model provider is configured, DocsAlive can draft a small fix and re-run the fixed page, retaining the fix only if it passes. Without a provider, triage is deterministic; fix pull requests open as drafts and are designed not to merge automatically.

README testing checklist

• [ ] List installation methods, supported versions and first-use paths.
• [ ] Start in a clean environment with the stated prerequisites.
• [ ] Run the actual documented install command and record its source.
• [ ] Verify the installed artifact, import or executable and version.
• [ ] Run complete blocks in reader order, without hiding missing setup.
• [ ] Check the meaningful result, exit status and relevant side effects.
• [ ] Explain placeholders, fragments and variable or omitted output.
• [ ] Record fixtures, credentials, interaction and platform coverage gaps.
• [ ] Test the candidate before release if the check is a release gate.
• [ ] Check the published install path and schedule checks for drift.
• [ ] Assign owners to failures and have a person review the reader journey.

What this does not replace

A test cannot decide whether the README teaches the right concept, explains a decision clearly or includes the example a new user needs. Ask a person unfamiliar with the project to follow the path, and pair execution with link, Markdown and prose checks. The CLI guide covers command-specific behavior; notebook testing covers executable documents with cells and saved output.

For CI setup, see testing docs in GitHub Actions. For site source formats and execution coverage, see testing code examples in static sites.

See pricing for DocsAlive plans: Team is $299 per month and Growth is $699 per month. To discuss coverage for your documentation, contact us.

FAQ

For Go-specific example functions and Markdown checks, see testing Go documentation examples. For Java and PHP READMEs, see testing Java documentation examples and testing PHP documentation examples. When README examples need an API key, see testing docs examples that need API keys.

For the install step itself, see testing installation instructions in docs.