Testing Python documentation examples

The short answer

Test Python documentation at the level of the promise it makes. Use doctest for interactive Python sessions and their expected output, pytest --doctest-modules to collect doctests from Python modules alongside a pytest suite, and a format-aware process for fenced examples in Markdown. Run the documented installation and examples in a clean environment, then check meaningful results. These approaches complement one another: a passing module doctest does not establish that every Markdown fence was found or that a public package installs correctly.

Start with the install and first useful example, then expand to supported Python versions and important alternatives. The release checklist provides a decision record; the JavaScript guide covers the equivalent questions for Node and TypeScript.

Use doctest for interactive sessions

Python's standard-library doctest finds examples written like an interactive interpreter session and compares the displayed result with expected text. For example, save this docstring in arithmetic.py:

def average(values):
    """Return the arithmetic mean.

    >>> average([2, 4, 6])
    4.0
    """
    return sum(values) / len(values)

Run python -m doctest -v arithmetic.py. The command imports and inspects the module, runs the example, and reports its comparison. Doctest is especially useful when the displayed representation is stable and is part of the lesson. It is also executable code: importing a module and running examples can have side effects, so run only trusted content in an appropriate environment.

The prompts and expected output are significant. >>> starts an input, ... continues one, and output follows without a prompt. A normal fenced Python block without interactive prompts is not automatically a doctest just because it is in a Markdown file. An exception example should show the expected traceback in doctest's supported form; avoid asserting incidental paths or line numbers that can vary.

Doctest compares text rather than understanding semantic equivalence. Representations, whitespace, ordering and floating-point output can differ between versions or platforms. Choose stable input and output, or use a narrowly scoped option such as ELLIPSIS when a portion is genuinely variable. An overly broad ellipsis can allow a wrong result to pass. For complex objects or nondeterministic output, an explicit assertion in a normal test may express the contract more clearly.

Docstrings should still be useful to a person reading them. Keep setup visible and examples small. If an example needs a file, create it in a controlled temporary location or explain where it comes from. Do not depend on a developer's current directory, network credentials or pre-existing data. If the documentation is an ordered tutorial, check the tutorial as a reader sequence too; independent docstrings do not establish the entire journey.

Collect module examples with pytest

With pytest installed in the active environment, run python -m pytest --doctest-modules from the project root. Pytest collects doctests in Python modules in its collection scope as well as its ordinary tests. Restrict the path when needed, for example python -m pytest --doctest-modules src/mypackage. Check the collection output and exit status: a successful run with no relevant examples is not evidence that the intended documentation was tested.

Collection can import modules and trigger optional dependencies or import-time behavior. Choose paths deliberately, install the dependencies required by the supported examples, and isolate side effects. If a project uses a src layout, ensure that the package installed in the environment is the intended one. An editable checkout is convenient for pull requests but can mask packaging omissions in a release artifact.

Pytest also supports doctest-style text files when configured or selected appropriately. That still does not mean arbitrary Markdown fences are executed: Markdown's language tags, prose and fences need a parser or a tool that understands the source format. Keep a separate inventory of module docstrings, doctest text files and Markdown pages. Specify which command covers each group.

Use fixtures and pytest configuration deliberately. Shared setup may make a test convenient, but if it supplies an import or variable omitted from a copyable example, readers cannot reproduce the pass. Put required setup in the example or clearly document its prerequisite. Review skipped doctests and collection errors as part of the release decision rather than assuming a green summary covers every module.

Test fenced Markdown examples

Locate the actual source of published pages, including included files, generated reference pages and versioned copies. Identify fences by language and intent: a complete Python program, an interactive session, an output block and an illustrative fragment need different treatment. Do not concatenate every fence blindly. A shell installation command is not Python, and two tabs may be alternatives rather than steps in one program.

For a complete standalone block, extract it with a Markdown-aware parser, write it to a temporary file and execute it with the intended interpreter. Preserve the page path and line number in the report. For dependent blocks, run the sequence in the order the reader sees it in a fresh process for that page. Capture exit status and output, then compare any promised result. Document whether a following text fence is expected output or merely an illustration.

Avoid a regular expression that silently loses nested fences or format-specific constructs. MDX components, includes and generated pages may need site-specific handling. Test the source that actually produces the published snippet, and inspect the rendered page for missing context or broken formatting. The static-site guide discusses these boundaries.

A fragment that intentionally omits setup can be syntax checked or tested with an explicit harness, but label the result accordingly. Keep the harness visible to reviewers. A test that adds import package cannot prove that the displayed block is independently copyable. If the page claims the example is complete, fix the page rather than silently filling the gap.

Isolate Python and dependencies

Create a new virtual environment for each meaningful test target. For example, python -m venv /tmp/docs-python followed by . /tmp/docs-python/bin/activate gives a Unix shell an isolated Python environment. Use python -m pip so installation and execution refer to the same interpreter. On Windows, use the activation command appropriate to that shell, or invoke the environment's Python directly. Run the documented installation command after creating the environment.

Record the Python version, operating system, dependency constraints and package source. Test a built wheel or other release candidate in a clean environment before publishing; testing the checkout alone can miss files absent from the distribution. Run outside the repository when a local source directory could shadow the installed package. After publication, test the public installation path separately.

Pin dependencies for reproducible checks, while also scheduling a representative run against the supported dependency range when appropriate. A lockfile makes a failure easier to reproduce but can hide a break affecting readers who install newer allowed versions. Keep the purpose of each run clear. Use temporary directories and controlled data, and clean up resources created by examples. Never run destructive or credentialed examples against production simply to make the suite pass.

Decide what a passing result means

A doctest checks selected interactive examples and their textual output. A module collection adds discovery through pytest, not coverage of every page. A Markdown execution harness checks only the blocks and variants it extracts. Installation, compatibility, output, side effects and prose each require an explicit decision. Inspect collection counts and skips, then use the cost of broken examples to prioritize the reader paths that matter most.

Compare output without hiding changes

Output can be a contract, an illustration or a partial excerpt. The page should tell readers which. A stable string or a required field can be compared exactly; a generated identifier, timestamp or unordered collection may need a narrower assertion.

Normalize only known variable fields. Removing every differing line can hide a renamed field or a missing error. Keep expected and actual evidence so a reviewer can decide whether the documentation, the implementation or the comparison should change. If the README shows a representative response, say which values are examples and which shape or fields the user should rely on.

Output also includes files, exit codes and side effects. A command that prints success but returns a failure status can break a script; a quiet command can succeed. Check the behavior the page describes rather than assuming that text on stdout is the only useful result.

Handle credentials, interaction and external state

Not every README example should run in the same unattended job. A command may require a secret, open a browser, wait for input, start a long-running server or modify a remote resource. Identify these cases and select an appropriate separate check. Do not put real keys in documentation, fixtures or logs.

A timeout is evidence to investigate, not an automatic verdict that the README is wrong. The process may be waiting for a prompt, an external service may be unavailable, or the command may be intended to remain running. Preserve the command and environment with the result. For credentialed behavior, use a separate authorized environment and a narrowly scoped test account when appropriate.

DocsAlive classifies examples needing credentials rather than executing them with real keys, including customer-provided test credentials. Interactive, graphical, destructive and long-running commands need a deliberate separate strategy; a skip or timeout does not establish whether the reader's full path works.

Put the result in context

A green run describes a particular set of examples, an environment and a point in time. Keep those details with the result: the documentation revision, artifact and version, operating system, runtime, dependency source, and commands used to set up the environment. Record the examples that were executed, checked without execution, skipped or left outside the selection. This makes the result useful when a reader reports a different outcome later.

Separate a broken example from a test infrastructure problem, but investigate both. A registry outage can block a valid install; an incorrect package name can look similar in a short log. Preserve the original error and identify the failing step. If a retry succeeds, retain enough information to see that the first attempt failed. Repeated intermittent failures can still affect readers even when a later retry works.

Give each failure an owner and a path to resolution. A documentation change may be the right fix when the product behavior is intentional; a product change may be needed when the documented behavior is the supported contract. If the release proceeds with a known gap, record the affected path, reason, owner and follow-up date. A hidden skip is not the same as an explicit decision to accept a limitation.

Make checks part of release work

Run fast, relevant checks when a change is proposed, and run the complete release selection against the candidate artifact before publishing. The GitHub Actions guide explains how pull request checks and scheduled runs answer different questions. Scheduled runs are useful for detecting changes in external dependencies or installation paths even when no documentation file has changed.

A release gate needs a named decision maker, an identified artifact and an agreed failure policy. If the suite only runs after publication it can detect an issue, but it cannot prevent that release. After publication, a separate check of the public package or endpoint can verify the path readers actually use. Preserve the distinction between candidate and published checks in the report.

A matrix should reflect supported promises rather than every possible combination. Start with the primary onboarding path and the oldest and newest supported runtimes where appropriate. Add operating systems, package managers or version combinations when the documentation promises them or past failures justify them. Show the matrix and its omissions to reviewers, so a single successful Linux run is not mistaken for proof of every platform.

Keep setup explicit. A fixture, mock server or prepared file may be necessary for a repeatable test, but it can also conceal a prerequisite that the page fails to explain. List what the harness supplies and compare it with what the reader sees. If a local fixture replaces a remote service, a pass establishes behavior against that fixture, not availability or compatibility of the live service.

Read reports and improve coverage

A useful report identifies the page, example, category, expected behavior, actual result and environment. Review skips and warnings alongside failures. A syntax check, a type check and an execution check establish different things; label each accurately. An example that runs and exits successfully may still return the wrong value or write a file somewhere unexpected, so check the behavior the surrounding text promises.

Avoid changing expected output simply to make a run green. First decide whether the product behavior is correct for the target release and whether the changed output affects the reader's task. Compare stable values exactly when possible. For timestamps, generated identifiers and unordered results, assert stable properties and explain variability. Normalize only what is truly variable, since a broad replacement can hide a meaningful regression.

Track coverage as an inventory, not just a percentage. Record the important paths that have no automated check, their owners and the reason. A credentialed example may need a separate authorized integration environment; an interactive setup may require a manual rehearsal. Assign an owner and review the list each release. Do not put real credentials in documentation, logs or public pull requests.

How DocsAlive fits

DocsAlive extracts supported fenced examples and runs or checks them in page order, with a fresh container for each page. It selects one variant from each tab group. Python, JavaScript on Node 20, simple TypeScript, shell and curl or HTTP examples have the coverage described in the framework reference. Compiled-language examples are recognized but not executed yet, and credential-dependent examples are classified rather than executed with real keys.

Read report.md and report.json with the findings reference. A pass can represent execution or a syntax check, and skipped blocks and unselected tab variants remain coverage gaps. A person still needs to review the explanation, the reader journey and any behavior the automated checks do not cover. When a model provider is configured, DocsAlive can draft a small fix and re-run the fixed page, retaining the fix only if it passes; fix pull requests are drafts and are designed not to merge automatically.

See pricing for DocsAlive plans: Team is $299 per month and Growth is $699 per month. To discuss coverage for your documentation, contact us.

For model-provider, agent and vector-database SDKs, see testing AI and LLM SDK documentation examples.

If your docs are built with MkDocs, see testing MkDocs documentation examples.

If your docs are built with Sphinx, see testing Sphinx documentation examples.