Testing AI and LLM SDK documentation examples

The short answer

Test AI SDK documentation in layers: start in a clean environment, install the documented package and extras, and run complete examples in page order. Check client construction, request arguments, response parsing, streaming and async control flow, and tool schemas. Use controlled responses for repeatable checks, then run a small, separately managed live suite to catch changes that mocks and recordings cannot see. Record skips and distinguish actual documentation defects from missing credentials, external failures, variable model output and intentional fragments.

Run fast offline checks on pull requests, test the release artifact before publication, and use scheduled checks to detect changes in providers and dependencies. A test that skips a live request has checked only part of the reader’s path. The SDK documentation testing guide covers the broader release workflow; this guide focuses on the additional moving parts in AI and LLM examples.

DocsAlive extracts supported examples, runs or checks them in page order, compares documented output where applicable, and classifies findings. It classifies examples needing credentials rather than executing them with real keys. Compiled languages are not executed yet, and TypeScript support is limited to simple cases.

Why AI SDK examples drift quickly

An AI example can depend on several independently changing interfaces. The SDK changes its client methods or parameters; a provider changes a response field; a model is retired; a vector database changes a query option; or an agent framework changes how it passes tool calls between steps. The code in the documentation may stay untouched while the reader’s environment changes around it.

A short RAG example can cross even more boundaries: an embedding client, a vector index, a retrieval result, a prompt template and a generation client. An evaluation example may add a dataset loader, a grader and a reporting library. A successful import of one package says little about whether the whole path still works.

These problems are related to documentation drift, but model-backed examples add a special concern: a working request may produce different language every time. A test needs to check the contract the page actually promises without requiring a model to repeat a particular sentence.

Define the reader’s path and the test boundary

Before automating a page, identify what a reader is expected to bring, install, run and observe. Keep complete blocks in page order when later blocks use variables, clients or files created earlier. Treat tabs as alternatives: executing a Python tab followed by a JavaScript tab is not the same as following either path.

For a tutorial spanning several pages, document the state carried between pages and test that journey separately. A fresh environment for each page is useful for finding hidden dependencies, but it cannot by itself prove a cross-page workflow. The quickstart testing guide explains how to check that sequence.

Make each test report the package artifact, setup and assumptions it used. If the test installs an extra dependency or supplies a fixture that the page does not mention, reviewers need to know. Otherwise a passing test can conceal a missing prerequisite.

For API reference examples, also check the relationship between the prose and the signature being exercised. A valid request is not enough if the description promises a field or behavior the example never verifies. See API reference documentation testing for that distinction.

Start with installation and request construction

Use a clean environment to find imports and extras that happen to be present on a maintainer’s machine. Test the command a reader is given, then import the modules used by the page. If a framework’s vector-store integration requires an optional extra, the page should identify it or state a clear prerequisite.

Run two kinds of installation check when they answer different questions. Installing from the checkout checks the code under review. Installing a built artifact in a fresh environment checks what the release contains. After publication, checking the published package tests the reader’s installation path. Record which one produced each result.

The examples below use a fictional acme_ai package, client and model identifier. They illustrate test structure; replace them with the actual SDK and its documented API in your own suite.

A request-building test can capture what the example sends without making a network call:

from types import SimpleNamespace

import acme_ai


def make_request(client):
    return client.generate(
        model="example-model",
        messages=[{"role": "user", "content": "Summarize this document."}],
    )


def test_documented_request(monkeypatch):
    captured = {}

    def fake_generate(self, *, model, messages):
        captured["model"] = model
        captured["messages"] = messages
        return SimpleNamespace(text="A short summary.")

    monkeypatch.setattr(acme_ai.Client, "generate", fake_generate)
    client = acme_ai.Client(api_key="example-key")
    response = make_request(client)

    assert captured["model"] == "example-model"
    assert captured["messages"][0]["role"] == "user"
    assert response.text

The fake has an explicit signature, so an unexpected argument in make_request raises an error. The test also checks the request shape the page intends to teach. The pytest monkeypatch documentation describes how the fixture restores changes after a test.

There is a boundary here: replacing the method means this test does not verify the real method’s signature or provider behavior. Add an unpatched construction or signature check where the SDK makes that practical, and use a separate live test for provider acceptance. Avoid making the fake so permissive that it accepts any arguments.

Test response shapes, streams and async paths

A response parser is part of the example’s public promise. If the page tells readers to access a nested field, test that access using a representative response. Include cases the page claims to handle, such as an empty result or a tool call, without inventing guarantees the SDK does not make.

For generated language, assert stable properties rather than an exact sentence. Depending on the page, those properties might include the presence and type of a text field, a list of citations, a score within a documented range, or a tool call with the required arguments. A nonempty response is a useful smoke check, but it does not establish factual accuracy or relevance.

For streaming, cover the loop itself. A test should show whether the documented code handles the event kinds it claims to support, combines text in the right order and reaches the end of the stream. If the SDK distinguishes content events from status or error events, include representative examples of each. A test that only creates a stream object has not tested the reader’s consumption loop.

Async examples need equivalent attention. Run the code in an async-capable test environment, check that calls are awaited, iterate with async for where required, and exercise the documented client cleanup. A synchronous fake for an async method can hide an incorrect await; use a fake with the same async behavior as the interface being demonstrated.

Tool or function calling deserves its own checks:

• Validate the schema’s required fields and types against what the page says the tool accepts.
• Check that the example parses returned arguments and passes them to the intended function.
• Test invalid or missing arguments if the page promises to handle them.
• Keep execution of consequential tools out of a documentation smoke test unless the test environment is explicitly prepared for it.

For RAG, use a small, known document fixture and verify the shape and handoff of retrieved results. Check that the generation step receives the intended context. For eval libraries, verify that records and scores reach the documented output structure. These are integration checks, not proof that the retriever finds the best evidence or that an evaluation measures the right quality.

Keep live calls separate from repeatable checks

A useful suite has a fast path that can run without provider access and a smaller live path that tests the external contract. The offline path can catch imports, request construction, parsing and control flow. The live path can catch authentication, provider acceptance, availability and response changes that local substitutes may miss.

Skip explicitly when credentials are absent

Use environment variables for credentials rather than placing keys in examples, repositories or fixtures. A test can declare its prerequisite with pytest.mark.skipif:

import os

import pytest
import acme_ai


@pytest.mark.skipif(
    not os.getenv("ACME_AI_API_KEY"),
    reason="requires an authorized test credential",
)
def test_live_example():
    client = acme_ai.Client(api_key=os.environ["ACME_AI_API_KEY"])
    response = client.generate(
        model="example-model",
        messages=[{"role": "user", "content": "Reply briefly."}],
    )

    assert isinstance(response.text, str)
    assert response.text.strip()

The skip reason belongs in the test report. A skipped test is not a passing live request, and it should not be counted as evidence that a model identifier or response shape currently works. Conversely, a missing key in a public pull request is not automatically a defect in the documentation.

Run live checks only with an authorized test account and a deliberate budget. Bound request size and frequency, and review rate-limit, quota and service errors separately from invalid documented code. A provider rejection can reveal a real defect, but the error and setup must support that conclusion.

Record HTTP interactions when replay is useful

VCR.py can record HTTP interactions in cassettes and replay them on later runs. This makes a representative response available to an offline test and reduces repeated calls. It can also make a parser test more realistic than a response assembled by hand.

Treat cassettes as fixtures with a lifecycle. Remove credentials and sensitive request or response data before storing them, review what is recorded, and ensure replay cannot silently make a new request in an offline check. Refresh recordings deliberately with an authorized account and inspect the changes. A recording is evidence of an earlier interaction; it does not prove the provider still returns that shape today.

A cassette and a small fake serve different purposes. A fake is easy to target at edge cases. A recording can preserve details of a real interaction. Neither should be the only evidence for a live integration that readers depend on.

Compare output at the right level

Python’s doctest options can make comparisons tolerant of limited variation. NORMALIZE_WHITESPACE ignores differences in whitespace, while ELLIPSIS lets an expected-output marker match a substring. Use them around the unstable part of otherwise meaningful output, not across the entire result: an ellipsis can match more than intended.

For model text, even a loose text comparison may encode an accidental expectation. Prefer explicit assertions on the structure and properties the page promises. If a sample answer is illustrative, label it as such. If the page promises exact output from a deterministic local transformation, compare that exact output.

Run offline checks in a clean container

A container can expose hidden dependencies and keep one page’s state from affecting another. Preserve state within a page when its blocks form a sequence, then start the next page in a fresh environment. Install dependencies during image preparation so the offline run does not need network access.

For example, after building an image named docs-check-image that contains the checkout and its test dependencies:

docker run --rm --read-only --network none \
  --tmpfs /tmp \
  docs-check-image \
  python -m pytest -p no:cacheprovider -m "not live"

--network none makes this an offline check; it cannot validate a provider call. --read-only means tests must write only to locations made writable for them, such as the temporary filesystem shown here. Configure any other required writable paths explicitly. The command is a starting point for isolation, not a complete security policy for arbitrary untrusted code.

Keep the image build and dependency provenance visible in CI. An image that already contains an undocumented integration package can make a page pass even when its install instructions are incomplete. A separate clean installation check should follow the reader’s documented steps.

Keep model identifiers current

A pinned model identifier makes an example reproducible only while that identifier remains available to the reader’s account and platform. Keep the identifier in a visible place, record why it was selected, and give the example an owner who can update the code and its surrounding explanation together.

Review the provider’s deprecation information on a schedule and before releases that depend on the example. OpenAI’s deprecations page lists affected APIs and recommended replacements. It also describes using the List models endpoint to inspect each model’s optional shutdown_date field: a date indicates an announced shutdown, while null indicates that no shutdown date has been announced. An absent or null value is not a guarantee of permanent availability.

Anthropic’s model deprecations page distinguishes active, legacy, deprecated and retired states. It also notes that partner-operated platforms can have different schedules. Check the platform your readers actually use rather than assuming one provider page covers every route to a model.

A model-list check is a signal for review, not a substitute for a request. Access can vary by account, and a listed model may not support every capability used in a particular example. When replacing an identifier, retest the request, response parsing, streaming or tool behavior the page demonstrates.

Build a pull-request, release and nightly plan

Use each CI trigger to answer a different question. Pull requests need fast feedback on the changed docs and code. A release gate should test the artifact intended for readers before publication. Nightly work can spend more time on dependency drift and authorized external checks.

The following plain GitHub Actions workflow illustrates the offline portion. It assumes the repository supplies a docs-check-image build through its Dockerfile, and that its test suite marks external tests as live.

name: Documentation examples

on:
  pull_request:
  release:
    types: [published]
  schedule:
    - cron: "23 9 * * *"
  workflow_dispatch:

permissions:
  contents: read

jobs:
  offline-examples:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository
        uses: actions/checkout@v4

      - name: Build documentation test image
        run: docker build -t docs-check-image .

      - name: Run offline example tests
        run: |
          docker run --rm --read-only --network none \
            --tmpfs /tmp \
            docs-check-image \
            python -m pytest -p no:cacheprovider -m "not live"

The release trigger above runs after publication. Use a separate pre-publication workflow or release process to build the candidate artifact, install it into a fresh environment and run the relevant checks before approving the release. The example workflow builds from the checked-out repository; it does not itself prove that the published package was installed.

Configure a separate, access-controlled live job when credentials are available. Give it a bounded set of tests and a budget, and make its report distinguish skips, provider failures and documentation failures. Do not expose secrets to untrusted pull-request code. The GitHub Actions docs testing guide covers CI design in more detail.

Triage failures before filing documentation bugs

The error message is the beginning of an investigation. A missing attribute may be a genuine SDK change, or it may result from installing the wrong package. A failed request may expose a retired model, or it may be a temporary external failure. Preserve the page, line, setup, artifact and error so a maintainer can reproduce the finding.

A classification is not a verdict on the whole page. A missing fixture may be reasonable in a reference example and confusing in a quickstart. An intentional fragment may still need clearer surrounding prose. A deprecation warning may deserve a planned update even when the example still runs.

Where DocsAlive fits

DocsAlive scans a local directory or public Git URL, detects supported documentation formats, extracts fenced code blocks and plans whether to execute, syntax-check or skip them. It keeps blocks in page order, recognizes tabbed alternatives and uses a fresh container per page. It can compare an output block shown immediately after code and reports the result with the page, line, status, category and reason.

Its statuses are pass, warn, fail and skip. A pass can mean execution or a syntax check, so read the report alongside the coverage and skip reasons. Its triage identifies likely documentation bugs such as api-drift, invalid-syntax, stale-output and package-not-found. It also distinguishes conditions including needs-credentials, network-external, missing-fixture, illustrative and depends-on-context. Runtime errors, timeouts and some external network results need review rather than automatic filing as docs bugs.

For AI SDKs, the credential boundary matters: the service classifies credential-dependent examples; it does not currently execute them with customer-provided test credentials. Use a separately managed live suite for those paths. Python, JavaScript and shell examples are among the supported execution paths; simple TypeScript annotations can be stripped, while full TypeScript and JSX are not judged. Compiled languages are recognized but not executed yet.

The service can run on pull requests, releases and nightly schedules. When a model provider is configured, it can draft a minimal fix and re-run the affected page; without one, triage is deterministic. Fix pull requests are drafts and require maintainer approval. See how docs testing works and the guides for Python, JavaScript and notebooks for related coverage decisions.

The published findings show why raw failure counts need triage. On one large open-source AI SDK documentation site, a scan of 261 pages extracted 1,327 code blocks and planned 927 for testing. Of those, 406 passed and 520 failed; triage flagged 7 likely docs bugs among the 520 failures. On another site, 45 blocks failed and none were escalated as docs bugs because they were fragments, credential-gated or TypeScript-only syntax. These are results from those scans, not a predicted rate for another site. See the published findings and how the pipeline works.

For a public repository you own or maintain, the health check provides a report from up to 40 documentation pages. The pricing page lists Open Source at $0 per month, Team at $299 per month, Growth at $699 per month and Scale from $1,499 per month, with different repository, page and run allowances.

What these checks do not replace

A successful example run does not establish that generated content is accurate, safe, unbiased or appropriate for a user’s purpose. It does not prove retrieval quality, evaluate an agent’s decisions across realistic tasks, or replace security review of tools and credentials. Those need their own evaluation datasets, threat analysis and human review.

Execution also cannot fully judge whether a page teaches the concept clearly, states prerequisites at the right time or explains an error a reader is likely to encounter. Review skipped blocks and alternative tabs, and have someone follow the complete journey. For teams comparing approaches, the free docs testing tools guide describes related options, while docs testing for API companies covers the wider maintenance process.

Pasteable checklist

• [ ] List the packages, optional extras, services, fixtures and credentials each example requires.
• [ ] Install the documented dependencies in a clean environment and record the artifact under test.
• [ ] Run complete blocks in page order and identify alternative tabs and intentional fragments.
• [ ] Check imports, client construction, request arguments and pinned model identifiers.
• [ ] Exercise response parsing, including the fields and types the page promises.
• [ ] Test streaming loops, async calls and cleanup where the page uses them.
• [ ] Validate tool schemas and the handling of returned arguments.
• [ ] Use controlled fixtures or cassettes for repeatable offline checks, with sensitive data removed.
• [ ] Skip credential-dependent tests with a visible reason when an authorized key is absent.
• [ ] Compare stable structure and properties instead of requiring identical generated text.
• [ ] Run selected live checks with an authorized account, bounded usage and a clear owner.
• [ ] Review provider deprecation information and the relevant platform’s model availability.
• [ ] Run fast checks on pull requests, test the release artifact before publication and check for drift nightly.
• [ ] Triage failures using the error, setup and reader context before filing documentation bugs.
• [ ] Review coverage gaps, page clarity and the end-to-end reader journey.

The sources we checked

These sources were read on November 9, 2026.

FAQ