Testing notebooks and executable docs

The short answer

Test a notebook as an ordered, executable document: start a fresh kernel, run its cells from top to bottom with the documented environment and inputs, and check the results that matter to the lesson. Decide whether saved output is part of the promise, distinguish stable results from variable displays, and record skipped cells, external dependencies and the exact environment under test.

Run focused checks on pull requests, test the intended release artifact before publication when a release gate is needed, and repeat execution on a schedule to catch dependency and data drift. Notebook-oriented tools such as nbval and nbmake can help execute notebooks as part of a test workflow; choose and configure a tool for the behavior and output comparisons you need. For the wider practice, see how to test documentation code samples.

DocsAlive supports execution or checking of certain fenced examples in documentation pages. The existing product coverage does not establish native notebook execution or saved notebook-output comparison; use a notebook-aware runner for those checks. Its credential, language and other coverage limits still apply to supported extracted examples.

Why executable documents drift

A notebook combines explanation, code, execution state and displayed results. That makes it a useful teaching format: a reader can change a value and see what happens. It also creates more ways for the published lesson and the behavior of the code to disagree.

A cell can depend on a variable created in an earlier interactive session but absent from the saved document. Cells can have been run out of order, so the displayed output reflects a sequence a new reader cannot reproduce. A dependency update can change an import, a dataset can move, or a plotting library can render the same data differently. The notebook may open normally while its first meaningful calculation fails.

Saved output adds a second timeline. The code in a cell may have changed since the output was last generated. A chart or table can look plausible even though it comes from an older dataset or package version. Conversely, a fresh run can produce a different timestamp or random identifier without contradicting the lesson. A useful check needs to distinguish these cases instead of declaring every difference either a bug or harmless noise.

Executable documentation comes in several forms: notebooks, pages generated from notebooks, and prose with runnable code blocks. Choose the unit of execution accordingly. A check of extracted Markdown code can be useful for a published page, but it does not necessarily run the original notebook with its kernel, cell metadata and stored outputs. Track these as separate pieces of evidence.

Inventory the notebook's promises

Begin with the reader's goal. What environment is required? Which cells must run in order? What data or services are needed? Which result should the reader recognize at the end? List the notebooks and the paths through each one before choosing a runner.

Prioritize the onboarding notebooks and examples tied to a release. Record an owner and an intended test environment for each. A notebook requiring a GPU or a paid external service may need a different job from a small local tutorial. If it is not run on every pull request, keep the gap visible and schedule an appropriate check rather than implying that a green suite covers it.

Use a person to review whether the lesson makes sense as well. A notebook can execute perfectly while its prose describes the wrong goal or its chart has an unreadable label. Execution is evidence about the path tested, not a replacement for editorial or scientific review.

Start a fresh kernel and run in order

The basic reproducibility test is simple: open the saved notebook in the intended environment, start a fresh kernel and execute cells from the beginning in document order. A fresh kernel removes variables and imports left behind by a maintainer's previous work. It also exposes hidden dependencies between cells.

Suppose one cell creates a small dataset:

values = [2, 4, 6]

A later cell uses it:

mean = sum(values) / len(values)
print(mean)

The expected result is 4.0. Running the second cell alone in a fresh kernel fails because values is undefined; running the saved document in order should establish the state. If the notebook presents the second cell as a standalone example, include the necessary setup or explain the dependency. Do not fix the test by injecting values outside the document without making that setup visible.

An execution count or stored display is not proof that the saved source will reproduce the result. Re-run the saved document in a clean kernel and retain the test result. If a notebook contains independent branches, mark them clearly and test the intended branches separately. A cell intended only for interactive exploration may need a different treatment from a required step in the tutorial.

Check working directory and files

Notebooks often assume a particular working directory. A relative path may resolve on the author's machine but fail when a reader launches the notebook from the repository root or when a CI runner starts elsewhere. State the expected launch location and test it. Record files the harness provides and their paths.

Use small, representative fixtures where practical. A fixture should have the columns, types and edge cases necessary for the lesson, and its provenance should be clear. If the published tutorial expects users to download a dataset, a local fixture tests the computation but not the download path. Test or review that path separately and label the distinction in the report.

Write generated files to an isolated temporary location where possible. Check that the notebook produces the expected file and that cleanup does not remove anything outside the test's own workspace. If a notebook intentionally modifies a shared resource, replace that action with an appropriate isolated target or exclude it with an explicit reason and an owner.

Reproduce the kernel and dependencies

A kernel name alone is not a complete environment specification. Record the language runtime, relevant package versions or constraints, kernel setup, operating system and any system libraries needed by the notebook. A reader should know how to create the environment before executing the first cell.

Test the installation instructions as well as the cells. Installing additional packages in the CI job can hide a missing prerequisite in the notebook. If the runner must supply something not intended for readers, explain why and record it. Separate a failure to create the environment from a failure in the lesson itself; both can block the reader, but they have different owners.

A pinned environment can make a baseline repeatable, while periodic checks against newer supported dependencies can expose upcoming drift. State which job uses which versions. A passing run against one environment is not evidence that every supported operating system, runtime or dependency combination works. Choose a matrix based on the support promise and document the combinations left out.

If the notebook uses an SDK, distinguish the checkout from a built candidate and a published package. For a release gate, install the candidate artifact in a fresh environment and execute the notebook before publishing. A later check against the published package verifies distribution at that time but cannot block an earlier release. The SDK docs release-gate guide describes the same distinction for examples more broadly.

Choose a notebook-aware test workflow

Notebook-focused tools can execute saved notebooks under a test runner. Tools such as nbval and nbmake are examples of this approach. Their options and output-comparison behavior differ, so consult the relevant tool documentation and configure the check explicitly instead of assuming that every tool compares every saved display in the same way.

A minimal workflow has five decisions:

1. Discover the notebooks that are part of the supported documentation, including generated or copied versions if those are separately published.
2. Create the intended environment and make the required kernel available.
3. Execute each selected notebook from a fresh kernel in document order, with appropriate time and resource limits.
4. Check failures and the stable results promised by the lesson, using saved-output comparisons only where the chosen tool and configuration support them.
5. Preserve a report with the notebook, failing cell, environment, inputs, skip reasons and actual evidence.

A successful execution shows that the selected cells completed under those conditions. It does not automatically establish that every displayed number is correct, every plot is legible, every dataset is current or every interactive branch was exercised. Add assertions or targeted comparisons for the claims that matter, and review output visually when presentation is part of the lesson.

For example, a notebook that teaches a transformation might assert the row count, column names and a few representative values after the transformation. A notebook that demonstrates a plot might verify the data and labels separately and still need a person to inspect the image. Avoid a test that merely confirms that a cell produced some output when the page promises a specific result.

Test the published form too

If a site renders notebooks into HTML, identify the source of truth and the generation step. Execute the notebook source, build the page, and check that the generated page includes the intended code and output. A passing source notebook does not prove the deployed page was rebuilt, and a rendered page that looks correct does not prove the source still executes.

Record the source commit, artifact and version associated with each build. When notebooks are distributed for download, check that the linked file is the intended version. If the website strips outputs, hides cells or changes execution during rendering, document those choices and verify the result readers actually receive.

A generated Markdown page may contain fenced examples that a documentation checker can inspect. That can complement a notebook run, but the resulting page may omit cell state, metadata or rich output. Do not equate a pass on extracted blocks with a successful fresh-kernel notebook execution.

Decide what output means

Notebook output can include text, tables, images, rich HTML, warnings and error displays. Decide what each example promises before choosing exact comparison, normalization, assertions or human review. Keep the raw actual result available when possible.

An exact comparison is appropriate for a stable value whose exact form matters. It is often too brittle for a timestamp or a plot rendered on different platforms. At the same time, broad normalization can conceal a real regression. Removing all numerical differences would hide a changed calculation; ignoring an entire table could miss a renamed column. Document each normalization and keep it as narrow as the claim permits.

When a notebook saves output, decide whether the repository expects it to be refreshed and reviewed. An executed test can pass while committed output remains stale if the workflow does not compare or update it. Conversely, a comparison can fail because a display library changed formatting while the underlying result is correct. Inspect code, actual output and the intended promise before changing either the notebook or the test.

Randomness and numerical tolerance

For examples using random data, choose whether reproducible values are necessary for the teaching goal. A fixed seed can make a run more repeatable, but record the runtime and library versions as well; a seed alone is not a guarantee of identical behavior across all implementations. For a statistical demonstration, a range or invariant may be more meaningful than an exact sample.

Floating-point calculations can differ slightly across environments. Use a justified tolerance when the lesson permits it, and retain an exact check for quantities that must be exact. A tolerance should reflect the problem, not simply be widened until a failure disappears. If a result changes beyond the expected variation, investigate the data, implementation and dependency versions before refreshing the saved output.

Errors and warnings

Some lessons deliberately demonstrate a failure. Make that intent clear and test the expected condition in a controlled way. An unexpected exception in an unrelated cell should not be treated as a successful demonstration. Similarly, a warning may be an expected teaching point, an environmental notice or a sign that a dependency will break in a future release. Preserve enough context to tell which.

A runner that stops on an unexpected error may leave later cells untested. Record that coverage gap in the report. A configuration that permits errors to continue needs equally careful review: completion alone does not mean all required cells succeeded.

Control data, secrets and side effects

A notebook can download data, query a service, start a server, create files or train a model. Before automation, classify these operations and choose an environment in which the intended behavior is safe and meaningful. Never commit real credentials or print them into saved output or CI logs.

Treat a skip as a documented gap with an owner and reason. A notebook may need a separate scheduled integration job, a manual check or a smaller teaching example. A fixture can make a local check reliable, but it should not be presented as evidence that an external service remains available or its data has not changed.

Inspect stored output before publication as well as execution logs. A secret or private data value can be captured in a display even when the source cell contains only a placeholder. Use appropriate access controls and cleanup for the environment, and keep test accounts and datasets separate from valuable production state.

Schedule checks around changes and releases

Notebook checks can be slow, so split them by the question they answer rather than abandoning automation. Run the first-use notebooks on relevant pull requests; run a broader suite on a schedule and before a release where its result can still influence publication.

Trigger relevant checks for changes to notebooks, package code, dependency definitions, data fixtures and the documentation build. A notebook can fail after a library change even if its file was untouched. Make ownership and expected response clear: a stable calculation regression may block a release, while a remote outage may need investigation and a separate decision about publication.

Keep runtime and cost visible. A small, representative smoke suite can provide fast feedback while a fuller suite exercises more notebooks, hardware and data. Explain which checks block and which report for review. Monitor scheduled jobs themselves; a green result from last month is not evidence that this week's scheduled run occurred.

Decide what a failure means

Read the failure with its environment and the promise being tested. A missing kernel, invalid import, changed numeric result and unavailable dataset require different responses. Retain the notebook path, cell, actual result, dependency versions, input provenance and runner logs.

Avoid refreshing all outputs blindly after a dependency update. Review meaningful changes with the person responsible for the lesson or result. A repeatable wrong answer is still wrong, and a visually different plot may still represent the same data. Conversely, an output mismatch can reveal a real change that would mislead readers.

Where DocsAlive fits

DocsAlive extracts supported fenced blocks from documentation pages, keeps blocks in page order and executes or checks supported examples in a fresh page container. It can compare an output block immediately following an example where applicable and classifies findings. Its reports include reasons for skips and other results; read them with the findings reference.

Those capabilities do not establish native .ipynb execution, kernel management or comparison of saved notebook outputs. Use a notebook-aware runner for that work. If a generated documentation page contains supported fenced examples, checking those examples can provide additional evidence about that page, subject to the usual coverage limits. It does not reproduce a notebook kernel or verify rich outputs merely because the source was a notebook.

Credentials are classified rather than used to execute examples with real keys, including customer-provided test credentials. Compiled-language examples are not executed yet, and TypeScript support is limited. A pass can represent a syntax check rather than execution; selected tab variants and skips also limit coverage. Review the framework reference before deciding which checks a report can support.

Notebook testing checklist

• [ ] Inventory supported notebooks, owners and the reader's intended result.
• [ ] Document runtime, kernel, dependencies, launch location and inputs.
• [ ] Start a fresh kernel and execute required cells in document order.
• [ ] Record fixtures and setup supplied by the test rather than the page.
• [ ] Check stable results, meaningful tolerances and expected error paths.
• [ ] Review saved output and generated pages for staleness.
• [ ] Identify random values, variable displays and narrow normalization rules.
• [ ] Isolate side effects and keep secrets out of source, output and logs.
• [ ] Record skips, hardware needs, external services and untested branches.
• [ ] Test the release candidate before publication when using a gate.
• [ ] Run focused pull request and scheduled checks, and assign failures.
• [ ] Review explanations, visual output and the full reader journey by hand.

What this does not replace

Execution cannot establish that a lesson is clear, that a dataset is appropriate for the question or that a chart communicates its result responsibly. Review the prose, data and visualizations alongside the test evidence. Use link and prose checks as complements; the free tools guide covers additional layers. For the first install and usage path in a repository, see testing README code examples.

For CI setup, see testing docs in GitHub Actions. For site source formats and execution coverage, see testing code examples in static sites.

See pricing for DocsAlive plans: Team is $299 per month and Growth is $699 per month. To discuss the coverage your documentation needs, contact us.

FAQ