Testing Sphinx documentation examples

The short answer

Test Sphinx documentation examples according to how they are authored. Keep Sphinx's doctest builder for doctests in reStructuredText pages, and run the linkcheck builder separately for external links. For Markdown pages, including MyST Markdown, use a scan that actually reads those files and inspect the fences it plans. A documentation build, an external link check and a code execution check answer different questions.

DocsAlive does not detect Sphinx or read .rst files. In a Sphinx project with no other recognized framework configuration it falls back to plain Markdown mode and reads .md, .mdx and .markdown pages. Plain python and pycon fences can be run; a MyST directive fence that opens with {code-block} followed by python is treated as text. It does not run Sphinx or its builders. Choose plain fences when execution by the product is the goal, and retain the native Sphinx checks for the rest.

Start with the docs testing overview and how it works. The Python guide covers the language-specific environment, while the TypeScript guide explains a different type-checking boundary. For a ReadMe platform export, see ReadMe documentation testing.

Identify how the code reaches the page

Sphinx supports several ways to show highlighted code. The directives reference distinguishes reStructuredText literal blocks, doctest blocks, the code-block directive and literalinclude. A doctest block shows an interactive Python session; the other approaches can show other languages. The choice affects both the source to review and the test that can find it.

A literal block places the text in the document. A highlight directive sets the highlighting language for subsequent literal blocks until another such directive appears. The setting is presentation information: it does not establish that the text runs or that its output is correct. A long page may change highlighting partway through, so identify the applicable setting before interpreting an unlabeled block.

The code-block directive accepts a language, and sourcecode and code are directive names for the same purpose. It gives authors control over individual blocks, including presentation options. That is useful when a page contains examples in multiple languages. A block can look convincing and still reference an outdated import or omit an installation step. Review the source and test the reader's sequence as well as the presentation.

literalinclude draws text from another file. The referenced file is normally relative to the current document, while an absolute path is relative to the top source directory. Options can select portions of that file. Record the included source and the selection when planning a test: executing a whole file does not necessarily test exactly the excerpt shown to a reader, and a correct excerpt can still depend on setup omitted from its surrounding page.

Inventory each form separately. For every page, note the authoring file, any included source, the language, the required environment and the result promised. Also identify generated or assembled content your site uses, and confirm which source supplies each example. The framework guide explains what the product detects, and the MkDocs guide offers a comparison for a different source format.

Use Sphinx builders for their own checks

The Sphinx builder reference describes the doctest builder as running the documentation's doctests when the sphinx.ext.doctest module is enabled in conf.py. Select it with sphinx-build -b doctest, providing the source and output directories. This is a Sphinx check, using the project's Sphinx configuration. Its result is evidence about the doctests it runs, not proof that every displayed code block was executed.

The linkcheck builder checks the integrity of external links. It answers a different question from the doctest builder. A working external link does not demonstrate that an example using the linked library works, and successful doctests do not establish that external links remain intact. Keep the results separately visible so the person reviewing a change can identify the relevant problem.

The sphinx-build reference explains that the command reads configuration from the source directory and selects a builder with -b. A normal HTML build produces documentation; it is another useful check of the site. Inspect the generated result when a change affects formatting, included text or navigation. None of these separate signals should be described as an end-to-end test of every reader journey.

Understand Python doctest output

Python's doctest module finds interactive examples beginning with >>>, runs them and compares their output with the expected output shown in the text. That comparison makes an example both an instruction and an assertion about the displayed result. Check that the expected output is stable and actually important to the reader. A changed value may be a real behavior change; incidental variation needs a deliberate treatment rather than a blind update.

A doctest is most useful when its inputs, setup and expected result are clear. Keep the example small enough to understand, but do not remove context that a reader needs. If a snippet relies on earlier state, record the intended sequence and verify it under the same assumptions as the test. A doctest result speaks to the examples and conditions it covers, not to unmarked shell commands or arbitrary illustrated fragments on the page.

Choose Markdown fences deliberately

The MyST code reference describes ordinary fenced code with a language identifier for highlighting. It also describes a directive-style code block with options such as a caption or emphasized lines. Those forms can both present code, but their source text is different. The product takes the first word of a fence's info string as its language, so {code-block} is not recognized as python.

This fictional page excerpt uses plain fences. The package is illustrative; use a real package and its documented setup in your own project.

# Work with the demo

```python
from acme_docs_demo import greeting
message = greeting("reader")
print(message)
```

```pycon
>>> 2 + 2
4
```

DocsAlive plans supported blocks in page order in a fresh container for each page. It runs Python fences and executes pycon sessions line by line, comparing printed output. The import above requires a real installed package to succeed; a fictional name makes the illustration clear, not an executable dependency. If a test harness supplies setup, record it beside the test so a missing instruction for readers does not become invisible.

A directive fence is useful when its presentation options are needed, but the product treats that fence as text and does not run it. A plain fence is the choice to make when the same source block should be selected by this scan. If a directive or included file must remain, keep it and add a separate appropriate test of the reader-visible content. Check the actual report to confirm which file, line and language were planned.

A generic CI outline

Run these commands from a repository root with Sphinx, the project dependencies and the product CLI already available. Replace the directories with the real source and build paths. The doctest module must be enabled in the Sphinx configuration for the doctest builder to run doctests.

sphinx-build -b html docs build/html
sphinx-build -b doctest docs build/doctest
sphinx-build -b linkcheck docs build/linkcheck
docsalive scan . --execute --include 'docs/**' --out docs-test-report

The scan only reads eligible Markdown files in its docs directory. If the source directory differs from the detected directory, use --docs-dir deliberately and check the include pattern. The CLI reference explains selection, --setup, and reports. A scan without --execute extracts and plans rather than executing, so make the execution choice explicit in CI.

Read the HTML build, doctest, linkcheck and scan results as separate evidence. Record the revision, dependency setup and any failures or skips. Run the checks on a documentation change and when the underlying package changes; repeat them later to find drift. If a check requires a service or credential, identify a separate safe environment for that path instead of assuming a local example run covers it.

Make coverage visible

A mixed Sphinx repository often contains .rst pages, Markdown pages and code in separate files. List the sources each check covers before interpreting a green result. The product does not extract indented literal blocks, reStructuredText code-block directives, literalinclude content or Sphinx doctest blocks. It does not use the Sphinx configuration, toctree, directives or builders. It reads eligible Markdown source directly, without rendering the site.

For .rst doctests, retain the Sphinx doctest builder and inspect its result. For included source, test the relevant file and selected excerpt with a suitable project-owned check. For Markdown, inspect the product report for executed blocks and omissions. For every remaining item, name an owner and a way to check it; this is especially important for a prominent installation or first-use path.

A page can also contain several dependent steps. The product preserves page order for planned blocks, but a page container is fresh. Explicitly document prerequisites and do not assume variables or files from another page exist. For a journey that spans pages, write a separate test that follows the full sequence and keeps the necessary state. The guide index and free tools guide can help identify complementary checks.

Review the source-to-result chain

A useful review starts at the page a reader sees and works backward. Identify the source page, the code as displayed, any file supplying included text and the expected outcome. Then work forward from the repository through the relevant build or test. This chain makes it clear whether a failure belongs to the source, the inclusion, the example's dependencies or the output shown on the page.

When the same code is included in more than one page, inspect each surrounding explanation. One page may introduce an environment variable or file that another page never mentions. A standalone check of the shared source is valuable, but it cannot prove that both pages give readers the context needed to use the excerpt. Conversely, a correct page introduction does not prove that the included source is current. Keep both checks when they answer different questions.

Doctest output needs similar care. Treat expected output as part of the documentation contract when the page relies on it to explain behavior. Compare the actual result in the intended environment and ask whether a difference changes the reader's understanding. For values that naturally vary, provide an explanation and choose a test that checks the meaningful property. Preserve enough evidence for a reviewer to decide why a changed output is acceptable.

A Sphinx project can also contain Markdown that is not part of the intended published documentation. A source scan and a site build can therefore have different scopes. Select the docs directory and include patterns intentionally, compare the report with the site's source inventory and explain any differences. The fact that a file was eligible for scanning does not alone show that it appeared in the built site; a build result and source report should be read together.

For a release, identify the package or artifact against which the examples should run. A development checkout may expose an API that is not in the package readers install. Repeat the relevant checks against the intended artifact, record the setup and verify a representative new-reader path. When the project changes dependencies, investigate differences before changing expected output or weakening a check. This makes a result useful to both the documentation author and the package maintainer.

Review the plan before focusing on the final count. A skipped block can be a deliberate fragment, an unselected alternative or unsupported content; each has a different implication. Note whether the excluded example is prominent, what a reader would need to run it and who owns its separate test. An inventory that lists these boundaries is more useful than a single green indicator with no explanation of what was omitted.

Keep the evidence current

Give each important check an owner and a trigger. A documentation change can alter the instructions; a package release can change an import or output without changing the page. Revisit the inventory for both kinds of change, and inspect the first failure before deciding how much downstream evidence remains useful. Record the artifact, setup and revision with the result so another maintainer can reproduce it.

A small representative check is a useful beginning, but prominent reader paths deserve complete coverage. Include setup, the action and the expected result, then add separate checks for alternatives or environments that matter. When a path remains untested, say what is missing and why. Revisit that decision when the documentation or its audience changes.

Triage failures before changing a page

A failing block is a starting point for investigation. Read the actual error, the documented prerequisites, the selected block and the environment together. A missing package in the test environment may point to an omitted installation instruction, an intentional prerequisite, or an incorrect setup choice. Reproduce the smallest reader journey before deciding which explanation fits.

A timeout, runtime error or external network result can need human review. An output difference may reflect a changing timestamp or ordering, but it may also reveal a change that matters to readers. Keep the expected and actual output with the finding. Do not update a result merely to turn a report green; establish what the page promises first.

The report makes these decisions easier to trace. Its JSON findings identify the page, line, language, status, category and reason, with a hint and evidence where supplied. The summary includes a count of likely documentation bugs. A pass can mean execution or a syntax check, while warn, fail and skip each require their own interpretation. Read the plan and the reason rather than treating all statuses as equivalent.

A reproducible fix should identify the installed artifact and any setup provided outside the page. Run the relevant check again after changing the source, and look for the next failure in the same sequence. If a page depends on an earlier page, test that longer journey separately: a fresh environment per page makes missing local setup easier to spot, but it does not establish the whole cross-page path.

Checklist for a documentation example test

The docs testing checklist helps turn that inventory into a repeatable review. The installation testing guide is useful when the first failure is in setup. For the wider relationship between builds, sources and reader-visible pages, see static-site documentation testing.

What this does not replace

DocsAlive does not detect Sphinx, read .rst files, extract Sphinx directives or run sphinx-build and the doctest builder. It does not render the site or check links. Keep the Sphinx builders and checks for included sources alongside Markdown example execution.

Human review still matters for whether the instructions are understandable, the output is meaningful and the reader can complete the task. A successful isolated block does not prove that a whole tutorial is coherent. Keep build, link, accessibility and visual review wherever they apply, and assign an owner to every important path the automated report did not cover.

For a starting point, the free health check lets you send a public repository you own or maintain; the team runs the engine on up to 40 of its docs pages and emails the real report within 2 business days; the checkout is deleted after the run. Pricing is Open source $0/month, Team $299/month, Growth $699/month, and Scale from $1,499/month.

The sources we checked

FAQ

Does a Sphinx HTML build run every code example?

A build produces the documentation; it is not evidence that every displayed snippet executed. Use the doctest builder for configured doctests and separate execution checks for other examples.

What is needed for the doctest builder?

Enable sphinx.ext.doctest in the Sphinx configuration and run the doctest builder with the appropriate source and output directories. Its result covers the doctests it runs.

Does linkcheck run Python code?

Its stated job is to check external link integrity. Keep Python execution and link checking as distinct results.

Can the product read an .rst page?

No. It reads eligible Markdown files and does not extract reStructuredText literal blocks, directives, includes or doctest blocks.

Will a MyST directive fence be run as Python?

No. A fence beginning with {code-block} is treated as text. A plain python fence is recognized as Python; select the form that meets the page's presentation and testing needs.

How are pycon sessions handled?

They run line by line and printed output is compared. Review the actual and expected output when a result differs, especially when a value can vary between runs.

Does a scan cover a file included by Sphinx?

It does not extract literalinclude content. Identify the included source and selected lines and test the relevant content separately.

Should a page depend on a previous page's state?

If it does, say so and test the complete journey separately. The product uses a fresh container per page, while keeping planned blocks in order within that page.