Testing docs in GitHub Actions

The short answer

Run documentation example checks in GitHub Actions when relevant changes arrive in a pull request, and run them again on a schedule to catch changes outside the repository. Start with a clean runner, install the dependencies that readers need, execute the selected examples, and publish enough context to diagnose a failure. Keep the check's scope and skipped examples visible.

A workflow can build a site, check links and execute examples, but these are different checks. A successful build does not establish that a command works. A link checker cannot tell whether the response example is accurate. Choose an execution tool or repository script for your examples, give it a clear exit status, and use the workflow to run it consistently. See testing code examples in static sites for source formats and extraction decisions.

DocsAlive extracts supported fenced examples and runs or checks them in page order. Reports distinguish findings and skips; credential-dependent examples are classified rather than executed with real credentials, compiled-language examples are not executed yet, and TypeScript support is limited. Inspect what was actually covered before interpreting a passing report.

Decide what the check promises

Start with a reader journey that matters: installing the package, making the first request against a safe fixture, or running a short example from the documentation. Write down the pages and language variants in scope, the runtime and dependency versions, and the expected result. A check should answer a precise question such as “Does this Python quickstart run against the candidate package?” rather than “Are the docs good?”

A useful result also names what it did not exercise. Some blocks are partial fragments, output samples or intentionally invalid examples. Others require interactive input, private data or a service you cannot safely reach from a pull request. Mark those cases explicitly and give them an owner. If the execution tool reports skips, read them alongside failures and passes. A green check with many skips has a narrower meaning than one with complete execution coverage.

The repository may contain a README, generated reference pages, notebooks and a static site. These have different lifecycles. The README guide covers installation and first use; the notebook guide covers kernels, cells and saved output. Use the same CI principles for each, but select an appropriate runner and assertions for its format.

A pull request and scheduled workflow

The following example assumes your repository provides scripts/check-doc-examples.sh and a pinned requirements-docs-test.txt. The script is your own test entry point: it should select examples, run them, print useful diagnostics and exit nonzero when a required check fails. Replace these paths and the setup with your project's actual files. The workflow does not depend on an invented DocsAlive command.

Save it as .github/workflows/docs-examples.yml:

name: Docs examples

on:
  pull_request:
  schedule:
    - cron: '23 4 * * *'
  workflow_dispatch:

permissions:
  contents: read

jobs:
  examples:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - name: Check out source
        uses: actions/checkout@v4
        with:
          persist-credentials: false
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'
          cache: pip
          cache-dependency-path: requirements-docs-test.txt
      - name: Install test dependencies
        run: python -m pip install -r requirements-docs-test.txt
      - name: Run documentation checks
        run: bash scripts/check-doc-examples.sh

The pull request event gives contributors feedback before a change is merged. A scheduled run can expose a moved dependency, changed public endpoint or broken installation path even when nobody touched the repository. workflow_dispatch provides a manual run. Scheduled workflows use the default branch, and GitHub can delay scheduled jobs during heavy load; the time in the cron expression is UTC. Pick a time away from the top of the hour when practical and treat the schedule as a monitoring signal, not an exact timer.

The script should be executable locally as well as in CI. A small wrapper that calls the same underlying commands in both places makes a failure easier to reproduce. Pin dependencies deliberately, including the test tool itself, and record the interpreter version. A separate scheduled job can exercise a broader range of supported versions or intentionally unpinned dependencies when you want early warning about upstream changes.

Run an initial version without filtering pull requests, observe its cost and coverage, then tune it. An example check often depends on application code, package metadata, fixtures and workflow configuration as well as documentation files. A filter containing only docs/** would miss an API change that breaks unchanged examples.

Use path filters with care

GitHub Actions supports paths and paths-ignore under events such as pull_request. If you use them, include all files that can change the result: documentation, shared snippets, the code and package definitions being demonstrated, the test harness, dependency locks and the workflow itself. Adapt the patterns to your repository. This example is an alternative trigger block, not a second on block to paste into the previous workflow:

on:
  pull_request:
    paths:
      - 'docs/**'
      - 'README.md'
      - 'src/**'
      - 'scripts/check-doc-examples.sh'
      - 'requirements-docs-test.txt'
      - '.github/workflows/docs-examples.yml'
  schedule:
    - cron: '23 4 * * *'
  workflow_dispatch:

A filter is an optimization, not a dependency graph. Shared configuration, generated files, an SDK in another directory or a change in a remote service may still matter. Review the pattern when the repository moves. For a monorepo, separate checks by component only when you can identify the dependencies between each documentation set and the code it describes. A broad scheduled run remains useful even when pull request checks are selective.

There is a practical branch protection consequence: a workflow skipped by path filtering can leave its required check pending, blocking a pull request. Test the rules with both matching and nonmatching changes before making that workflow a required status check. A common alternative is to trigger the workflow on every pull request and perform conditional work inside it, with an always-present aggregation job that reports the decision. Design and test that logic carefully so an unexpectedly skipped test cannot be mistaken for a pass.

Filters on branches and paths can combine, narrowing the events that run a workflow. Pull request events also differ from push events; choose the events that match your review process and verify the actual checks on a test pull request. Scheduled runs are not made selective by the pull request paths in this example. If a release candidate needs a gate, add a separate trigger and test the candidate artifact before publication instead of assuming the default-branch schedule covers it.

Choose the version under test

The checkout on a pull request represents proposed repository content, but that alone does not say which package the examples use. If the test imports an already installed release, it can miss a breaking change in the proposed code. If it installs from the checkout, it may miss files omitted from the built distribution. Make the choice visible in logs and in the check name.

For a pull request, install or build the proposed source using the same packaging path you intend to ship. For a release gate, build the candidate artifact and install that artifact into a fresh environment before running the documentation examples. After release, a separate check can install from the public registry and catch publication, naming or availability problems. These runs answer related but different questions; a successful local source test does not prove the published install command works.

Avoid allowing setup scripts to hide missing reader steps. If an example needs a fixture, document the fixture as test setup. If the page promises an install command, run that command in a suitable isolated environment rather than silently substituting a development install. Record substitutions and platform limitations in the report. A matrix can add supported Python versions or operating systems, but only add combinations you can maintain and investigate when they fail.

Cache dependencies without caching confidence

actions/setup-python@v5 can cache pip's download cache, as in the first workflow. The dependency file helps determine the cache key. This speeds up installation; it does not replace installation or verify that the result matches what a reader can install today. Keep the install step and pin your test dependencies where reproducibility matters. Use a lock file or other dependency manifest supported by your project's tooling, and update it deliberately.

Cache keys need to reflect inputs that change dependencies, including runtime version and lock files. If you use actions/cache directly, choose keys that include those inputs and follow its documented restore behavior. Do not cache virtual environments blindly across incompatible operating systems or interpreter versions. Do not store credentials, private keys or sensitive generated data in caches. Cache contents may be available to pull request workflows under GitHub's cache access rules, so assume untrusted contributors could read accessible cache entries.

A warm cache can conceal a registry outage or an unavailable package. Consider an occasional clean install with caching disabled if the public installation path is part of the promise. That run may be slower and more exposed to network failures, so report the installation stage separately. A dependency cache is a performance aid; it is not a substitute for a clean environment or a reason to skip a public-package check.

Protect secrets and runner permissions

Pull requests can contain changes to scripts and examples that the workflow executes. Treat that code as untrusted, especially when it comes from a fork. Use the ordinary pull_request event with minimal permissions for checks that execute contributor code. Do not use pull_request_target to check out and execute untrusted pull request code with elevated access or secrets. GitHub restricts secrets and token permissions for forked pull requests in common configurations, but repository settings and event choices matter; do not make safety depend on a single assumption.

Set the minimum token permissions at workflow or job level. The example requests only contents: read and disables checkout credential persistence. Avoid granting write access just to run tests. Review actions and scripts as dependencies, and consider pinning third-party actions to full commit SHAs according to your organization's policy. Version tags in the example make the action versions readable; immutable pinning provides a stronger guarantee about the code that runs.

Use fake values and local fixtures for untrusted pull request checks. If a separate trusted scheduled or release job must access a service, give it a narrowly scoped, revocable credential for a dedicated test account and limit its access and lifetime. Never put a real token in documentation, a workflow file, a cache or an artifact. Avoid printing environment variables, request headers and response bodies that may contain secrets. Masking is helpful but does not make arbitrary logs safe; transformations and derived values may escape masking.

Put networked tests behind explicit controls. Restrict their targets to an authorized sandbox, bound their time and cost, and clean up created resources. A dry run or mock can cover client behavior without proving a real service integration. Name that distinction in the report. If a credential is exposed, revoke and rotate it according to your incident process rather than relying on deleting the log.

Report results that people can act on

A failing job is useful only if an owner can locate and reproduce the failure. Print the source page, line or block identifier where available, language, command or test name, exit status and a concise error. Include the package version, runtime and relevant fixture version. Avoid dumping the entire environment or raw secrets. A report should distinguish a failed assertion from an extraction error, a network outage, an unsupported block and an intentional skip.

Use the workflow log for immediate diagnosis, and write a small machine-readable report if your checker supports one. Upload sanitized diagnostics as an artifact when it helps investigation, with a retention period appropriate for your organization. Ensure reporting still runs after a failed check if you add an upload step, and verify that the job remains failed when the underlying test fails. A reporting step should never turn a failed execution into a green required check.

A job summary can list tested paths, versions, counts of passed, failed and skipped examples, plus a link to detailed logs. The counts need definitions: a syntax-only check is different from execution, and a skipped example is neither a pass nor a failure. Include any examples that were not discovered because of unsupported formats or selectors when your tool can report them. Review a sample report with a documentation author who did not write the test.

For flaky external checks, preserve the first failure and its context. Retrying can distinguish a transient service problem from a repeatable defect, but unlimited retries can hide a real regression. Assign ownership and a response target for scheduled failures; otherwise a nightly red badge becomes background noise. For a pull request, decide which checks block merging and which provide advisory information before adding branch protection.

Separate the layers of CI

A site build catches rendering problems and, depending on configuration, warnings about missing references. A link checker tests whether selected destinations resolve. An example runner executes selected code or checks its syntax. A human review assesses whether the explanation and outcome make sense. Run these as separately named steps or jobs so the team can see which promise failed.

External link checks can fail because a remote site is rate limited or temporarily unavailable. Execution can fail because a dependency changed or a test service is down. Neither automatically means a documentation edit is wrong. Record enough evidence to classify the failure, fix the underlying problem, and keep the test enabled when possible. Suppressions should include a reason, owner and review date rather than becoming a permanent list of ignored failures.

Start with a few critical paths, then expand based on incidents and reader traffic. Add checks when a new example is introduced, and remove or revise checks when the documented contract changes. Review execution time, coverage and failures periodically. The goal is reliable feedback that maintainers can use, not a large number of green checks whose scope nobody understands.

Review changes to the workflow itself

The workflow, dependency files and scripts are part of the test system. A pull request that changes them can change what gets tested or what a passing status means. Review these files with the same care as production code. In particular, look for removed assertions, broadened exclusions, changes to token permissions, newly introduced network destinations and steps that suppress errors. An ownership rule can help route such changes to maintainers who understand the check.

Keep a small set of sample inputs for the extractor or harness when those tools are maintained in the repository. Include a known passing example, a known failing example and an intentional skip. Periodically confirm that the harness identifies all three correctly. This check validates the reporting path as well as execution: an ignored failure or a missing block should be visible before it affects a real pull request.

A workflow that uses a matrix should give each combination a recognizable name and report its own version and platform. If one combination fails, investigate that combination instead of assuming every environment has the same problem. Set realistic timeouts so an unresponsive process does not consume a runner indefinitely. Limit concurrency where runs are expensive, but ensure cancellation of an obsolete run does not leave its result presented as a completed pass.

Keep a short maintenance note in the repository with the workflow file, commands for local reproduction, fixture setup, known exclusions and the expected owner. When a scheduled check fails, include the run link and the first meaningful error in the issue or internal tracking process. Decide whether a failure requires a documentation correction, a product fix, an updated dependency or an infrastructure repair. That classification helps the team address the cause rather than merely restarting the job until it turns green.

Finally, review the workflow after major changes to repository layout, package publishing or supported runtimes. A renamed directory can make a path filter incomplete while the YAML remains valid. A new installation method can require an additional check. A removed runtime can leave a matrix testing a contract the project no longer offers. Treat the workflow as maintained documentation of the checks themselves.

A practical rollout

First, run the checker locally against one self-contained page and confirm it fails when you deliberately break an assertion. Next, put that same command in the workflow and open a test pull request. Verify the check appears, logs locate the failing block, and the job status changes when the issue is fixed. Exercise a nonmatching pull request if using path filters, and test how branch protection treats the resulting status.

Then add a scheduled run and an owner for its alerts. Test a clean install periodically and decide whether a separate trusted integration job is necessary. Add platform or version combinations only after the first path is stable. Finally, document how to reproduce the check, where fixtures live, what skips mean and who updates dependencies. A new maintainer should be able to interpret the result without reverse engineering the workflow.

The workflow in this guide is generic infrastructure; it does not assert a particular integration or command for DocsAlive. When a model provider is configured, the product can draft a small fix and re-run the fixed page, retaining the fix only if it passes. Without a provider, triage is deterministic; fix pull requests are drafts and are designed not to merge automatically. Review all proposed changes and the coverage behind the report.

See pricing for DocsAlive plans: Team is $299 per month and Growth is $699 per month. To discuss coverage for your documentation, contact us.

FAQ

For the same coverage decisions on GitLab, see testing docs in GitLab CI.