Testing ReadMe documentation examples

The short answer

A ReadMe documentation workflow needs checks for the repository, OpenAPI descriptions, the published experience and the examples a reader actually follows. Git can serve as the source of truth, with pull requests for review and the appropriate ReadMe workflow for publishing or synchronization. Linting and validating an API description are valuable, but neither proves that an example still works with the package or service a reader uses.

DocsAlive detects a ReadMe export from rdme.yml or .readme; otherwise it falls back to plain Markdown. It reads eligible .md, .mdx and .markdown pages, plans supported code fences and treats recognized <Tabs> groups as alternatives, choosing one variant. It also parses OpenAPI files as a structural check. It does not run either ReadMe CLI, publish content, render the hub or test the Try It! console.

Begin with the docs testing overview, how it works and the framework reference. For a different authoring system, see the Sphinx guide; for typed snippets, see the TypeScript guide. The README testing guide concerns repository README files, a different subject from the ReadMe platform.

Map the source and publishing workflow

The docs-as-code guide describes maintaining documentation in Git as the source of truth, reviewing changes through pull requests and publishing with rdme. The bi-directional sync guide describes synchronization between the repository and ReadMe. Choose the workflow that matches your project and identify which branch and files are actually used. An example test against a stale branch can be technically successful while saying little about the page readers see.

The rdme guide describes syncing Markdown and OpenAPI definitions, including pre-upload validation. Its publishing workflow is distinct from the CLI used for a repository using bi-directional sync. Document the route your project uses, the review point and the person responsible for publishing. A test report should identify the revision it examined so it can be related to the reviewed and published content.

ReadMe supports MDX, and its MDX reference describes built-in components including Tabs. Tabs can present choices to a reader. Determine whether the choices are complete alternatives or whether later instructions depend on the selection. A test that follows only one tab cannot establish that every language, installation method or request example works.

For the product scan, recognized <Tabs>, <CodeGroup>, <RequestExample> and <ResponseExample> groups are alternatives: one variant is picked and the others are skipped as alt-variant. Consecutive code fences outside a recognized tab component are each planned on their own. Do not assume that adjacent blocks form alternatives merely because a reader might choose between them. Inspect the plan, and arrange independent coverage for important paths that were not selected.

Use the ReadMe CLI checks deliberately

The ReadMe CLI reference describes @readme/cli for projects using bi-directional sync. It runs against a cloned repository and does not itself upload content; publishing happens on merge. Its lint command checks frontmatter, slugs, links, content and structure. A clean lint run exits 0; warnings alone also exit 0, while an error produces a nonzero exit. Read warnings even when CI reports success.

oas:validate checks JSON or YAML OpenAPI documents at the top level of reference/. It takes no file argument and does not check specifications in subfolders there. It exits 0 when every specification is valid and 1 when any has an error. The --dereference option also resolves each $ref. The CLI reference says lint runs the same validation, so decide how you want to present the results without treating repeated checks as independent proof of runtime behavior.

oas:sync reads OpenAPI documents at the top level of reference/, creates pages for operations, updates changed endpoints and removes pages for deleted endpoints. Review and commit generated changes. The reference says it can generate pages from a document that parses but fails validation, and skips a file it cannot parse as YAML or JSON. Validate before syncing and inspect the generated diff rather than treating page generation as validation.

dev starts a local preview with hot reload. Use it to review how a change appears and how the page reads in context. A preview provides useful presentation evidence, but it does not prove that an external service accepts a request or that an SDK example executes. Similarly, a valid OpenAPI description establishes properties of the description, not whether your code samples still run.

Separate API descriptions from live behavior

An OpenAPI document can describe operations, parameters and examples. Validating and resolving references can reveal structural problems before publishing. It cannot by itself show that the deployed service matches the document, that a credential has the expected permissions or that a particular sample produces the claimed result. The API reference testing guide helps organize those checks around what readers need.

DocsAlive parses OpenAPI files found in the repository, resolves each $ref, and counts operations and examples. This is a structural check. It does not send requests to the live API from the OpenAPI file or compare the description with the running service. OpenAPI reference checks are listed on the Growth plan. Keep a separate authorized integration check when the relationship between a description and a live service matters.

A code fence containing a request is a different input from an OpenAPI document. Supported curl and HTTP blocks can be sent with curl, but examples requiring credentials are classified rather than executed with real keys; customer-provided test credentials are not currently supported. Record this limitation for authenticated paths, and use an appropriate separate test environment and credentials under your own control. Avoid publishing real keys in either documentation or reports.

Write a page as a reader journey

The following fictional excerpt illustrates two possible ways to make a request. The host and package are placeholders, not a working service or published dependency. The examples intentionally use an environment variable instead of a real key. Treat the blocks as separate paths when designing your own tests.

# Inspect the demo

Set `ACME_API_KEY` in your shell before using the service.

```bash
curl --header "Authorization: Bearer $ACME_API_KEY" https://api.acme-docs-demo.test/items
```

```python
import os
from acme_docs_demo import Client

client = Client(api_key=os.environ["ACME_API_KEY"])
print(client.list_items())
```

The Python import represents the fictional acme-docs-demo package. A real page should explain installation, credentials, the expected response and any necessary test data. A reader should be able to tell whether the two blocks are alternative ways to perform one task or consecutive steps. As written, they are consecutive fences without a recognized tab wrapper, so the product plans each on its own; it does not infer an alternative group.

Place real alternatives inside a recognized tab component if that matches the page design, and check which variant was selected. Even then, a single selected variant is partial coverage. Test the other paths from equivalent clean starting conditions when they matter. A curl request and an SDK call can fail for different reasons, including an API change, SDK change or missing prerequisite.

Run and inspect the scan

The product uses a fresh container per page and preserves the order of planned blocks. It can execute supported Python, JavaScript, simple TypeScript, shell and request examples; JSON and TOML are validated. Some languages are recognized without execution. The report records the result and reason for each block, making omissions visible as well as failures. The CLI guide explains options for setup, selection and output.

The scan reads eligible files under the detected docs directory: docs/ when present, otherwise the repository root, or a directory selected with --docs-dir. Match the chosen directory to your export and inspect the plan before claiming coverage. A file under another directory, an unsupported block or a skipped tab remains a separate item in the inventory.

A generic CI outline

For a bi-directional sync repository, the following commands illustrate checks from the repository root. The needed tooling and project setup must already be available. oas:sync changes files, so run it locally after a specification change and review its output before committing rather than treating it as a read-only CI check.

npx @readme/cli lint
npx @readme/cli oas:validate --dereference
docsalive scan . --execute --include 'docs/**' --out docs-test-report

Adapt the pattern and directory to the actual repository. Review report.md and report.json, including skips and hints. A plan without --execute does not run the examples. Run the relevant checks before merging, review the generated pages and preview, and confirm the intended publishing or synchronization path after approval. For a project using rdme to push content, follow its documented workflow rather than assuming these bi-directional sync commands are interchangeable.

Assign separate coverage for each promise

Start a coverage record with the source file, the reader-visible page and the operation or task it teaches. Add the selected tab, package or API artifact, prerequisites and expected outcome. Then list which tool checked each item. A lint result, a validated description, a running snippet and a visual preview each support a different conclusion. This record makes it possible to fix the right source without claiming that one check covers the whole experience.

When an example depends on a service, agree on what test data and permissions are needed. The fictional host above should never be mistaken for a real endpoint. In a real project, keep secrets outside the page, use a suitable controlled environment, and make the expected response precise enough to distinguish a genuine regression from changing data. The installation guide covers a common prerequisite, and JavaScript documentation testing covers runtime examples.

Review changes before they reach readers

A pull request can change prose, a code fence, an OpenAPI description or a generated reference page. For each change, identify which source is authoritative and which other files are derived from it. Review the generated diff alongside the underlying change, especially when an operation has been added, changed or removed. A valid description and a generated page can still convey an instruction that is incomplete for a real reader.

A practical review record includes the repository revision, branch, relevant source paths, CLI results, scan report and the preview decision. If a test setup installs a dependency or creates a fixture outside the page, record that too. This makes it possible to distinguish what the reader is told to do from what the test supplied. It also helps reproduce an issue after a package or service changes.

For tabbed examples, list the reader paths separately. A Python path and a shell path may show the same operation but use different packages, quoting rules and setup. Check that each path introduces its prerequisites, produces the intended response and leads sensibly into the next instruction. Since the product chooses one variant, a reviewer should know which was chosen and how the remaining important choices will be covered.

Where an example shows a response, decide which fields are stable enough to compare. A changing identifier or timestamp may need a different assertion from a fixed status or a documented field. Do not replace expected output simply because the observed response changed. First check whether the service, test data or page promise changed, and preserve a concise explanation with the review.

The publishing workflow also deserves its own check. Confirm that the reviewed branch and the intended ReadMe project correspond, then verify the page after the project's normal publication process. A local scan cannot prove that the resulting hub has the right navigation, layout or link destinations. A visual and editorial review can catch missing context even when every supported code block runs successfully.

Keep a visible backlog for unsupported languages, credential-dependent examples and unselected alternatives. Assign an owner and a suitable separate check to the reader paths that matter. The goal is not to eliminate every skip from a report; it is to know what each skip means and avoid presenting incomplete coverage as a complete test of the guide.

Keep the evidence current

Give each important check an owner and a trigger. A documentation change can alter the instructions; a package release can change an import or output without changing the page. Revisit the inventory for both kinds of change, and inspect the first failure before deciding how much downstream evidence remains useful. Record the artifact, setup and revision with the result so another maintainer can reproduce it.

A small representative check is a useful beginning, but prominent reader paths deserve complete coverage. Include setup, the action and the expected result, then add separate checks for alternatives or environments that matter. When a path remains untested, say what is missing and why. Revisit that decision when the documentation or its audience changes.

Triage failures before changing a page

A failing block is a starting point for investigation. Read the actual error, the documented prerequisites, the selected block and the environment together. A missing package in the test environment may point to an omitted installation instruction, an intentional prerequisite, or an incorrect setup choice. Reproduce the smallest reader journey before deciding which explanation fits.

A timeout, runtime error or external network result can need human review. An output difference may reflect a changing timestamp or ordering, but it may also reveal a change that matters to readers. Keep the expected and actual output with the finding. Do not update a result merely to turn a report green; establish what the page promises first.

The report makes these decisions easier to trace. Its JSON findings identify the page, line, language, status, category and reason, with a hint and evidence where supplied. The summary includes a count of likely documentation bugs. A pass can mean execution or a syntax check, while warn, fail and skip each require their own interpretation. Read the plan and the reason rather than treating all statuses as equivalent.

A reproducible fix should identify the installed artifact and any setup provided outside the page. Run the relevant check again after changing the source, and look for the next failure in the same sequence. If a page depends on an earlier page, test that longer journey separately: a fresh environment per page makes missing local setup easier to spot, but it does not establish the whole cross-page path.

Checklist for a documentation example test

The docs testing checklist helps turn that inventory into a repeatable review. The installation testing guide is useful when the first failure is in setup. For the wider relationship between builds, sources and reader-visible pages, see static-site documentation testing.

What this does not replace

DocsAlive does not run rdme or @readme/cli, upload or sync to ReadMe, render the hub, check links or test the Try It! console. It does not send live API requests from an OpenAPI file. Go, Rust, Java, Ruby and PHP are not executed; credential examples are classified rather than executed with real keys, and jsx and tsx are not judged. Keep independent checks for those paths.

Human review still matters for whether the instructions are understandable, the output is meaningful and the reader can complete the task. A successful isolated block does not prove that a whole tutorial is coherent. Keep build, link, accessibility and visual review wherever they apply, and assign an owner to every important path the automated report did not cover.

For a starting point, the free health check lets you send a public repository you own or maintain; the team runs the engine on up to 40 of its docs pages and emails the real report within 2 business days; the checkout is deleted after the run. Pricing is Open source $0/month, Team $299/month, Growth $699/month, and Scale from $1,499/month.

The sources we checked

FAQ

Does OpenAPI validation run the examples?

No. It validates the description. A separate example or integration test is needed to establish runtime behavior.

Which ReadMe CLI should I use?

The ReadMe CLI reference distinguishes @readme/cli for a bi-directional sync repository from rdme for pushing content. Follow the workflow configured for the project.

What does a zero lint exit mean?

A clean run exits zero, and warnings alone also exit zero. Inspect warnings rather than interpreting zero as proof that every issue or example is resolved.

Why was a specification missed?

oas:validate and oas:sync read OpenAPI documents at the top level of reference/, not its subfolders. Validation takes no file argument.

Are all tabs tested?

The product picks one variant per recognized group and skips the others. Record the choice and independently test other important reader paths.

Are adjacent code fences treated as tabs?

No. Consecutive fences outside a recognized tab component are each planned on their own. Make alternatives explicit in the page and the coverage plan.

Does the OpenAPI check call my service?

No. The product parses the description, resolves references and counts operations and examples as a structural check. It does not compare the description with the live service.

Does the product publish the page?

No. It does not run either ReadMe CLI, upload or sync content, render the hub, check links or test the Try It! console.