Testing code examples in MkDocs, Docusaurus and Sphinx sites

The short answer

Test the code examples in a documentation site by finding their source, identifying which blocks make a runnable reader journey, and executing those blocks in an environment that matches the instructions. Build the site and check links as separate layers. A successful MkDocs, Docusaurus or Sphinx build tells you something about rendering and configuration; it does not by itself prove that the examples work.

The extraction method must understand the formats and conventions in the project. Markdown fences, MDX components, reStructuredText directives, included files and generated reference material can all contain examples. Record the source location and language, preserve ordering and dependencies, and make intentional exclusions visible. Start with a high-value page and expand as you learn where the real examples live.

DocsAlive extracts supported fenced examples and runs or checks them in page order. Reports distinguish findings and skips; credential-dependent examples are classified rather than executed with real credentials, compiled-language examples are not executed yet, and TypeScript support is limited. Do not infer support for every MDX component, rST directive, site generator or build plugin from those fenced-example capabilities. Check the report and the actual format coverage for your project.

Locate the source of truth

A rendered page is often assembled from more than one source file. Before writing a checker, trace a few representative pages from the published URL back to their source. Find the repository directory, navigation configuration, include mechanism, generated content and any build plugins. Ask which file a contributor is expected to edit when an example changes. That is usually the best place to attach a diagnostic.

For MkDocs, documentation commonly lives in a Markdown source directory configured in mkdocs.yml; themes and <param> may add syntax, includes or generated pages. A fence in a Markdown file may represent executable code, sample output or a configuration file. Navigation settings describe the site's structure but may not enumerate every source file, so decide whether to test all source files or only pages included in a particular build.

Docusaurus projects commonly store documentation in Markdown or MDX files and configure document sets, versions and sidebars in the site configuration. MDX can combine Markdown with JSX components. Tabs, imported code, reusable components and custom transformations may place the visible example somewhere other than a simple fence in the page. Versioned documentation may repeat a page for several releases; a check needs to know which version of the software each copy describes.

Sphinx projects frequently use reStructuredText sources with directives for code blocks, literal includes, and other content. Sphinx can also process Markdown when an appropriate parser is configured. These <param> and generated API reference material can introduce examples from docstrings or external files. A literal include may point to a source file that is itself tested elsewhere; check the include's selected lines and the surrounding explanation as well as the file's independent test.

These are common patterns, not rules for every site. A project can store source outside the usual directory or produce documentation during its build. Inventory the actual repository and build configuration before deciding what a scanner covers. Create a small manifest mapping published pages, source files, included examples, applicable product versions and owners. This makes missing coverage visible when directories or generators change.

What a build proves

A documentation build parses sources, applies configured <param> and generates an output site. It can catch malformed markup, missing included files, broken internal references and warnings, depending on the generator and configuration. Run the project's documented build command in CI, and decide which warnings should fail the build. Review warnings when upgrading a theme or extension instead of disabling them indiscriminately.

Builds vary. A project may configure a strict build, suppress some warnings, or use plugins that run additional code during generation. A successful build therefore proves only that the configured process completed under those conditions. A syntax-highlighted block can be perfectly valid Markdown while containing a Python call to a function that no longer exists. A rendered output block may show a result that the current program no longer produces.

Build plugins and custom components are themselves code. Run builds from untrusted contributions with appropriate isolation and minimal credentials, just as you would run an example test. Keep build dependencies pinned where practical, and record the versions that produced a failure. A build can be a useful prerequisite for execution tests, but avoid silently assuming that every build executes every code block.

Build-time execution is possible when a project deliberately configures a tool or extension to execute examples while building. In that case, learn exactly which content it selects, what environment it uses, and how it reports failures. An extension that runs selected examples does not automatically cover other fences, tabs or generated pages. Consider whether executing during every local preview makes authoring too slow or unexpectedly contacts a service; a separate CI execution step can give clearer control.

Find examples in Markdown

A Markdown fence usually contains a language label and body. The label is useful for selecting a runner, but it is not a complete contract. A python block could be a complete program, a fragment requiring earlier imports, an illustration with placeholders, or expected output mislabeled as code. A bash block may include prompts or comments that are not intended to be copied. Identify the convention used by the project before executing anything.

Consider this simple sequence:

```python
from example_package import greet
print(greet("reader"))
```

Expected output:

```text
Hello, reader!
```

A useful test associates the output with the Python block and checks the stable promised result. It installs the appropriate package first and records which version was used. Merely parsing the Python fence finds syntax errors but does not establish that the import exists or the result matches. Conversely, executing the output fence as shell code would be a classification error.

Do not rely on a regular expression that recognizes only three backticks if the documentation uses longer fences, tildes, nested lists or extension-specific syntax. Use a parser compatible with the project's actual Markdown dialect, or explicitly document the narrower subset your extractor supports. Preserve source paths and line numbers where possible. Test the extractor against representative pages, including fences inside tabs, admonitions and included material.

A snippet can depend on an earlier block. Decide whether your checker runs blocks in page order with shared state, runs a complete example file, or requires each block to stand alone. Each choice affects what a pass means. Shared state can match a tutorial but can also hide an omitted import if setup is injected by the harness. An isolated block exposes missing prerequisites but may reject an intentionally incremental lesson. Write the execution contract next to the test configuration.

Account for MDX and components

MDX is more than Markdown with a different extension. A Docusaurus page can contain JSX elements, imported components, expressions and custom code presentation. A block nested inside a component might be visible in the generated page but inaccessible to a scanner that only reads top-level Markdown fences. A component can also render examples from data or a source file that a text search of the page never sees.

Trace one instance of each component used to display code. Determine where its content originates, whether tabs represent alternative languages or sequential steps, and whether the build transforms the content before display. A parser or build plugin that understands the project's MDX version can provide better structural information than scanning raw text. Treat imported or generated examples as separate sources when appropriate and attach diagnostics back to the page authors can edit.

Test all promised alternatives when feasible. A page with Python and JavaScript tabs does not become fully tested because the default tab ran. Record each selected tab, its language, runtime and outcome. If one variant is unsupported or requires a separate service, mark that gap explicitly. A screenshot of a rendered tab or a successful MDX compilation checks presentation, not whether the code inside works.

Component behavior matters too. A copy button can accidentally include prompts or omit required lines even if the underlying example executes in a test harness. For critical onboarding paths, inspect or test the rendered copyable content in addition to running the source. Keep that UI test distinct from the code execution test so failures point to the right layer.

Account for reStructuredText and includes

In rST, code can appear in directives such as code-block or literalinclude, and a literal block can follow a paragraph ending in a double colon. Indentation and directive options affect what appears in the output. An extractor that searches only Markdown fences will miss these examples. Use an rST-aware parser or the project's Sphinx processing pipeline, and verify its behavior with the <param> actually enabled by the site.

For a literal include, resolve the target relative to the source and honor the options used to select content. If the page displays only a portion of a file, running the full file can prove useful behavior while failing to prove that the displayed portion stands alone. The reverse is also true: a displayed fragment may be correct only in the context of a larger example. Record which file and lines were displayed, which code was executed, and any setup supplied outside the page.

Sphinx can pull API documentation from docstrings, depending on project configuration. Examples in those docstrings may need a separate discovery pass and a test environment with the package installed. Some projects already use doctest-style examples; inspect those tests before adding a second runner. Verify that the examples are part of the test selection and that output normalization does not erase the property the page promises.

Generated reference pages introduce a related ownership issue. Editing generated output may be overwritten on the next build. Find the generator input or source docstring, report the failure there, and regenerate the page through the project's normal process. Keep a small representative test fixture for custom directives and <param> so a parser upgrade cannot silently drop an entire class of examples.

Decide what can be executed

Classify blocks before running them. A complete example should have a language, prerequisites, inputs and an expected result. A fragment might demonstrate one line of a larger program and need a wrapper. Configuration examples may need a validator or a service that reads them. Shell output, deliberately broken examples and pseudocode should not be executed as if they were programs.

Use explicit metadata or a maintained manifest when the format supports it. Record whether a block is runnable, a continuation, an expected output, a negative example or an intentional exclusion. Include a reason for skips and an owner where a gap matters. Do not let a language label alone decide whether a network request, file deletion or package installation is safe to run. Review commands and restrict their execution environment.

Fixtures are part of the contract. Supply small, non-sensitive data with a known schema; create temporary directories and isolated test accounts when required. If a reader needs to create a file, include that step in the page or clearly distinguish it from test-only setup. Clean up resources after the test, including on failure where possible. Make remote calls only to authorized targets and limit cost, rate and duration.

For dynamic output, assert the stable properties that matter. An identifier or timestamp may vary, but the response shape, status and relevant values can still be checked. Normalize only documented variability. Broadly ignoring output, errors or ordering can make a stale example appear healthy. For negative examples, assert the intended error and its cause rather than treating any failure as success.

Build-time and runtime checks

Choose when to run examples based on feedback speed, safety and the promise under test. A build-time check can give authors immediate feedback and can verify content after configured transformations. It can also make local previews slow, require dependencies unrelated to writing, or introduce network calls during a build. Make these tradeoffs explicit in project instructions and keep a reproducible CI path.

A separate runtime check can install the product, set up fixtures and run selected examples after the site build or independently of it. This makes execution failures easier to distinguish from rendering failures and lets you use a different environment. It also creates a risk that the checker reads different source or versions from those published. Log the commit, source paths, package version and test configuration to connect the result to the site build.

For release documentation, test the candidate artifact in a clean environment before publication when the check is a gate. A test against a development checkout may miss packaging omissions. After release, separately check the public installation path if that is a reader promise. Versioned documentation may need tests against several supported releases, with old versions pinned to compatible dependencies. Retire or revise those checks deliberately when support changes.

Some examples are intentionally interactive or depend on hardware, a browser session or a paid external service. A mock or local substitute can cover part of the behavior, but it does not establish the real integration works. Keep a separate authorized integration test or manual procedure where appropriate, and describe the remaining gap. Do not provide live credentials to untrusted pull request code.

Link checks are a different layer

A link checker can discover broken internal paths, anchors or external destinations, depending on its configuration. It cannot tell whether the code on a reachable page executes or whether the text is accurate. An example runner can prove a selected command works under its test conditions, but it cannot guarantee that every navigation link leads to the intended page. Use both, together with the site build and editorial review.

Decide whether links should be checked in source, generated HTML or both. Source checks can identify the original file; generated output captures transformed URLs and links created by components. Exclude local-only or intentionally unavailable targets explicitly. External checks can encounter rate limits, authentication and temporary outages; distinguish those failures from a confirmed broken link and avoid turning transient noise into a permanent ignore rule.

Anchor behavior can depend on the generator, <param> and final rendered heading. Test representative internal links after the build when anchors matter. For external services, use reasonable timeouts and request rates, and follow the destination's access requirements. Review exclusions periodically with an owner and a reason. A link that resolves can still point to an obsolete version, so include critical destinations in human review.

Put the checks in CI

Run the site build, link checker and example runner as separately named steps or jobs. A pull request check should cover the source and product changes that can affect the selected pages, not only edits under the documentation directory. A scheduled run can catch dependency and external-service drift between commits. The GitHub Actions guide shows a generic workflow, path-filter tradeoffs, dependency caching and secrets hygiene.

For each failure, report the page or source file, block identifier or line, language, runtime, tested package version and concise diagnostic. Show whether the result was execution, syntax validation, an intentional skip or an unsupported format. Keep logs and artifacts free of credentials and private fixture data. A summary count is helpful only if the team understands what was discovered and what the checker could not see.

Begin with one high-value reader path and introduce a deliberate break to verify that the pipeline catches it. Add another page using a different source convention, such as an MDX tab or rST include, and confirm that the report points back to the editable source. Track coverage by page and variant. Expand gradually, assigning owners to failures and reviewing skips when the site, product or test tools change.

The README guide covers the repository's first install and usage path. The notebook guide covers cell order, fresh kernels and saved outputs. A site may include both, so treat them as related sources with their own execution requirements rather than assuming that the static-site build covers them.

A checklist for maintainers

• [ ] Map published pages to editable sources, included files and generated inputs.
• [ ] Identify Markdown dialects, MDX components, rST directives and enabled <param>.
• [ ] Inventory language tabs, versions, fragments and expected output blocks.
• [ ] Choose parsers and verify discovery on representative pages.
• [ ] Record the selected examples, source locations, setup and skipped cases.
• [ ] Run runnable examples with suitable dependencies and isolated fixtures.
• [ ] Assert meaningful results and keep normalization narrow.
• [ ] Build the site and check links as distinct checks with distinct reports.
• [ ] Test candidate artifacts before release when a release gate is required.
• [ ] Schedule broader checks, assign owners and review failures and exclusions.
• [ ] Have a person review the explanation, copyable content and reader journey.

Execution cannot establish that a tutorial teaches the right concept or that an example is the one a reader needs. It is one source of evidence alongside rendering, links, accessibility and editorial review. Use a failure to improve both the source and the test, and use a pass only for the scope actually exercised.

See pricing for DocsAlive plans: Team is $299 per month and Growth is $699 per month. To discuss coverage for your documentation, contact us.

FAQ

For MkDocs sites specifically, see testing MkDocs documentation examples. For VitePress sites, see testing VitePress documentation examples. For Docusaurus and Mintlify sites, see testing Docusaurus documentation examples and testing Mintlify documentation examples.

For Sphinx projects and ReadMe-hosted docs, see testing Sphinx documentation examples and testing ReadMe documentation examples.