Docs testing checklist for releases

The short answer

Before a release, test the code paths readers need against the artifact they will receive. Record the documentation revision, release candidate, environment, checks, gaps and decision. A useful checklist identifies who owns each item and what evidence is required; checking boxes without inspecting the output can create false confidence.

Use this list alongside the Python and JavaScript and TypeScript guides. The cost guide explains why onboarding and support paths deserve particular attention.

1. Define the release and its readers

• [ ] Identify the documentation revision and the exact candidate artifact or service version.
• [ ] Name the release owner and the person responsible for the docs test decision.
• [ ] List supported runtimes, operating systems, module formats and installation methods.
• [ ] Identify the primary install, quickstart, authentication, common task and recovery paths.
• [ ] Identify versioned pages, language tabs, generated examples and included source files.
• [ ] Record changes to APIs, defaults, dependencies, packaging and output that could affect examples.

The purpose is to connect each test to a reader promise. A page on the default branch can describe a future release while a public package remains older. Note the intended relationship and make version navigation clear. A source checkout may behave differently from a packaged candidate, so give each target a separate identity in the record.

Prioritize the first successful path and any examples changed by the release, then expand to supported alternatives and high-impact workflows. Check examples that depend on changed behavior even when the documentation itself did not change. A path filter limited to docs files can miss precisely that kind of drift.

2. Prepare a clean test environment

• [ ] Start from a fresh environment for each independent reader journey.
• [ ] Record the OS, runtime, package manager, dependency source and versions.
• [ ] Install the built candidate using the intended consumer path when possible.
• [ ] Run outside the checkout if local files or workspace dependencies can shadow the package.
• [ ] Follow the documented setup instead of relying on global tools, aliases or credentials.
• [ ] List fixtures, environment variables, configuration and extra setup supplied by the harness.
• [ ] Use controlled data and temporary resources with a cleanup plan.

A clean environment makes missing prerequisites visible. It does not mean every test must access the public internet or run against a live service. A controlled fixture can make a check repeatable; the evidence should say what it replaces. Compare setup with the page and fix omissions instead of silently providing a missing step.

Choose a matrix that matches the support statement. A primary operating system and runtime can provide quick feedback, but each additional supported version or platform needs a deliberate coverage decision. Record omissions and owners rather than implying that a single environment covers all users.

3. Exercise the documented path

• [ ] Run the install command exactly as presented, including package name and version constraint.
• [ ] Confirm the imported package or invoked executable is the candidate under test.
• [ ] Run dependent examples in page order and independent alternatives in fresh state.
• [ ] Verify imports, filenames, working directories, configuration and prerequisites.
• [ ] Check the result the prose promises, including exit status and relevant side effects.
• [ ] Compare stable output exactly and assert meaningful properties of variable output.
• [ ] Check expected errors and recovery instructions as well as the success path.
• [ ] Verify links, rendered formatting and whether placeholders are explained.

A command that exits successfully may still produce an empty result, use the wrong version or create a file in the wrong place. Check observable behavior against the claim. If the expected output changes, determine whether the product change is intentional and whether the explanation should change. Avoid broad normalization that could hide a real mismatch.

A code fragment can be useful without being independently executable. Label it as a fragment, explain its context and record whether it was syntax checked, type checked or exercised in a harness. A harness that adds missing imports or setup cannot establish that a supposedly complete copyable example works as published.

4. Review coverage and gaps

• [ ] Verify that the extractor found the intended files, blocks and variants.
• [ ] Review skips, warnings, collection errors and checks that only validate syntax or types.
• [ ] Identify examples requiring interaction, credentials, external services or unsupported runtimes.
• [ ] Record the owner and alternative verification for each material gap.
• [ ] Check generated pages and rendered tabs against their source.
• [ ] Ask a person to follow the primary journey and review clarity and missing context.

Counts are useful for spotting changes in discovery, but a count alone says little about importance. One untested installation path may matter more than many tested trivial snippets. Describe the affected reader journey and the consequence of a failure. The static-site guide discusses formats, includes and site builds.

Keep real credentials out of examples and logs. Use authorized test accounts and limited permissions for a separate integration test where necessary. If that is not available, mark the live path as unverified. A mock can test code against a known response but does not establish the availability or behavior of the real service.

5. Triage and decide before publication

• [ ] Preserve the failing page, line, command, output, artifact and environment.
• [ ] Distinguish a product regression, documentation defect and infrastructure problem.
• [ ] Assign an owner and confirm the intended behavior with the relevant maintainer.
• [ ] Re-run the affected path after a fix against the same candidate or a newly identified candidate.
• [ ] Record any accepted failure or coverage gap, its impact and follow-up date.
• [ ] Obtain the release owner's decision before publishing.

A registry outage and an incorrect package name can look similar, but their resolutions differ. Preserve the original failure and investigate. A retry may help diagnosis; repeated retries should not erase evidence that the reader path was unreliable. If a fix changes the artifact, record its new identity and repeat the checks needed for that candidate.

The decision should be explicit: block, fix and rerun, or accept a documented risk with an owner. A scheduled check after publication is valuable, but it is not a pre-release gate. The SDK release guide walks through the candidate and published artifact distinction.

6. Verify the public path and follow up

• [ ] Check that the public installation source serves the expected version after publication.
• [ ] Run the first-use journey against the published artifact in a fresh environment.
• [ ] Check links and version navigation on the published site.
• [ ] File and assign issues for any accepted gaps, with a review date.
• [ ] Schedule checks for dependency, service and documentation drift.
• [ ] Keep the evidence with the release so later reports can be compared.

Post-release verification can find registry, deployment or caching issues that a candidate test cannot. It is detection after the event, so have a response owner and a way to correct the page or package. Scheduled runs then help identify external changes between releases. The GitHub Actions guide covers automation choices.

Protect secrets and runner permissions

Pull requests can contain changes to scripts and examples that the workflow executes. Treat that code as untrusted, especially when it comes from a fork. Use the ordinary pull_request event with minimal permissions for checks that execute contributor code. Do not use pull_request_target to check out and execute untrusted pull request code with elevated access or secrets. GitHub restricts secrets and token permissions for forked pull requests in common configurations, but repository settings and event choices matter; do not make safety depend on a single assumption.

Set the minimum token permissions at workflow or job level. The example requests only contents: read and disables checkout credential persistence. Avoid granting write access just to run tests. Review actions and scripts as dependencies, and consider pinning third-party actions to full commit SHAs according to your organization's policy. Version tags in the example make the action versions readable; immutable pinning provides a stronger guarantee about the code that runs.

Use fake values and local fixtures for untrusted pull request checks. If a separate trusted scheduled or release job must access a service, give it a narrowly scoped, revocable credential for a dedicated test account and limit its access and lifetime. Never put a real token in documentation, a workflow file, a cache or an artifact. Avoid printing environment variables, request headers and response bodies that may contain secrets. Masking is helpful but does not make arbitrary logs safe; transformations and derived values may escape masking.

Put networked tests behind explicit controls. Restrict their targets to an authorized sandbox, bound their time and cost, and clean up created resources. A dry run or mock can cover client behavior without proving a real service integration. Name that distinction in the report. If a credential is exposed, revoke and rotate it according to your incident process rather than relying on deleting the log.

Review changes to the workflow itself

The workflow, dependency files and scripts are part of the test system. A pull request that changes them can change what gets tested or what a passing status means. Review these files with the same care as production code. In particular, look for removed assertions, broadened exclusions, changes to token permissions, newly introduced network destinations and steps that suppress errors. An ownership rule can help route such changes to maintainers who understand the check.

Keep a small set of sample inputs for the extractor or harness when those tools are maintained in the repository. Include a known passing example, a known failing example and an intentional skip. Periodically confirm that the harness identifies all three correctly. This check validates the reporting path as well as execution: an ignored failure or a missing block should be visible before it affects a real pull request.

A workflow that uses a matrix should give each combination a recognizable name and report its own version and platform. If one combination fails, investigate that combination instead of assuming every environment has the same problem. Set realistic timeouts so an unresponsive process does not consume a runner indefinitely. Limit concurrency where runs are expensive, but ensure cancellation of an obsolete run does not leave its result presented as a completed pass.

Keep a short maintenance note in the repository with the workflow file, commands for local reproduction, fixture setup, known exclusions and the expected owner. When a scheduled check fails, include the run link and the first meaningful error in the issue or internal tracking process. Decide whether a failure requires a documentation correction, a product fix, an updated dependency or an infrastructure repair. That classification helps the team address the cause rather than merely restarting the job until it turns green.

Finally, review the workflow after major changes to repository layout, package publishing or supported runtimes. A renamed directory can make a path filter incomplete while the YAML remains valid. A new installation method can require an additional check. A removed runtime can leave a matrix testing a contract the project no longer offers. Treat the workflow as maintained documentation of the checks themselves.

Put the result in context

A green run describes a particular set of examples, an environment and a point in time. Keep those details with the result: the documentation revision, artifact and version, operating system, runtime, dependency source, and commands used to set up the environment. Record the examples that were executed, checked without execution, skipped or left outside the selection. This makes the result useful when a reader reports a different outcome later.

Separate a broken example from a test infrastructure problem, but investigate both. A registry outage can block a valid install; an incorrect package name can look similar in a short log. Preserve the original error and identify the failing step. If a retry succeeds, retain enough information to see that the first attempt failed. Repeated intermittent failures can still affect readers even when a later retry works.

Give each failure an owner and a path to resolution. A documentation change may be the right fix when the product behavior is intentional; a product change may be needed when the documented behavior is the supported contract. If the release proceeds with a known gap, record the affected path, reason, owner and follow-up date. A hidden skip is not the same as an explicit decision to accept a limitation.

Make checks part of release work

Run fast, relevant checks when a change is proposed, and run the complete release selection against the candidate artifact before publishing. The GitHub Actions guide explains how pull request checks and scheduled runs answer different questions. Scheduled runs are useful for detecting changes in external dependencies or installation paths even when no documentation file has changed.

A release gate needs a named decision maker, an identified artifact and an agreed failure policy. If the suite only runs after publication it can detect an issue, but it cannot prevent that release. After publication, a separate check of the public package or endpoint can verify the path readers actually use. Preserve the distinction between candidate and published checks in the report.

A matrix should reflect supported promises rather than every possible combination. Start with the primary onboarding path and the oldest and newest supported runtimes where appropriate. Add operating systems, package managers or version combinations when the documentation promises them or past failures justify them. Show the matrix and its omissions to reviewers, so a single successful Linux run is not mistaken for proof of every platform.

Keep setup explicit. A fixture, mock server or prepared file may be necessary for a repeatable test, but it can also conceal a prerequisite that the page fails to explain. List what the harness supplies and compare it with what the reader sees. If a local fixture replaces a remote service, a pass establishes behavior against that fixture, not availability or compatibility of the live service.

Read reports and improve coverage

A useful report identifies the page, example, category, expected behavior, actual result and environment. Review skips and warnings alongside failures. A syntax check, a type check and an execution check establish different things; label each accurately. An example that runs and exits successfully may still return the wrong value or write a file somewhere unexpected, so check the behavior the surrounding text promises.

Avoid changing expected output simply to make a run green. First decide whether the product behavior is correct for the target release and whether the changed output affects the reader's task. Compare stable values exactly when possible. For timestamps, generated identifiers and unordered results, assert stable properties and explain variability. Normalize only what is truly variable, since a broad replacement can hide a meaningful regression.

Track coverage as an inventory, not just a percentage. Record the important paths that have no automated check, their owners and the reason. A credentialed example may need a separate authorized integration environment; an interactive setup may require a manual rehearsal. Assign an owner and review the list each release. Do not put real credentials in documentation, logs or public pull requests.

How DocsAlive fits

DocsAlive extracts supported fenced examples and runs or checks them in page order, with a fresh container for each page. It selects one variant from each tab group. Python, JavaScript on Node 20, simple TypeScript, shell and curl or HTTP examples have the coverage described in the framework reference. Compiled-language examples are recognized but not executed yet, and credential-dependent examples are classified rather than executed with real keys.

Read report.md and report.json with the findings reference. A pass can represent execution or a syntax check, and skipped blocks and unselected tab variants remain coverage gaps. A person still needs to review the explanation, the reader journey and any behavior the automated checks do not cover. When a model provider is configured, DocsAlive can draft a small fix and re-run the fixed page, retaining the fix only if it passes; fix pull requests are drafts and are designed not to merge automatically.

See pricing for DocsAlive plans: Team is $299 per month and Growth is $699 per month. To discuss coverage for your documentation, contact us.

For versioned upgrade paths and release notes, see testing migration guides and changelog examples.