Testing JSON, TOML and YAML examples in docs

The short answer

Test each complete configuration or data example at three levels: parse it as the format named by its fence, validate its structure against an appropriate schema when one exists, and check that the relevant tool or API accepts it at the version readers use. Parsing catches malformed syntax, but it cannot establish that a key is supported, a value is meaningful, or a request produces the documented response. Label intentional fragments and format variants clearly, run format-specific checks in CI, and review failures in context.

A JSON request body, a pyproject.toml file and a YAML deployment manifest make different promises. A reader may copy one into an API request, save another as a complete file, and use the third as part of a larger configuration. Your tests should reflect those promises and report what they actually checked.

DocsAlive extracts fenced examples and checks JSON and TOML syntax within the limits described below. It recognizes YAML fences but does not parse them yet, skips several related formats, and does not validate schemas or check configuration keys against a tool version. Pair it with format-specific checks and acceptance tests for the tools your readers use.

Why configuration examples break

Configuration blocks often look simpler than executable code, but they still depend on syntax rules, versions and surrounding context. A copied comment can make a JSON body invalid. An indentation change can alter the structure of a YAML document. A tool upgrade can rename a setting while leaving the old example perfectly parseable.

The fence label matters too. A block labeled json tells a reader and many checkers to expect JSON. If it contains JSON with comments or JSON5 syntax, the label describes a different format from the content. A syntax highlighter may make the block look plausible while a strict consumer rejects it.

Some failures are version-specific. A configuration can be valid TOML and still use a key that an older release of a tool does not understand. Conversely, a newer release may remove a formerly supported key. Record the format version and the tool version separately so a passing result has a clear meaning.

Documentation drift affects configuration just as it affects imports and commands. Test the examples when the docs change, when the relevant software releases, and on a schedule if dependencies can change independently. The broader problem is covered in documentation drift.

Decide what the reader is supposed to copy

Before testing a block, decide whether it represents a complete file, a complete request or response body, or a fragment. A complete block that readers are told to copy should pass the checks appropriate to its stated purpose. A fragment can be useful, but it should say what surrounds it and how to incorporate it.

For example, this is a complete JSON object with a replaceable string value:

{
  "project": "sample-project",
  "region": "west"
}

The values are illustrative, but the object is syntactically complete. Whether project and region are accepted by a particular API is a separate question.

By contrast, the following is deliberately broken JSON. The ellipsis stands for omitted content and is not a JSON value:

{
  "project": "sample-project",
  ...
}

Introduce a block like that as a fragment and explain where it belongs. If readers need to copy it, provide a complete companion example or instructions for replacing the omitted content. Do not silently remove placeholders during testing and then imply that the displayed block itself passed.

Use three levels of checking

The three levels answer different questions. A useful report says which levels ran, against which files or blocks, and what was skipped.

Parsing is the first gate. It can find a missing quote or malformed inline table without installing the application. It is fast enough to run on every documentation change, but it only checks syntax as implemented by the selected parser.

A schema can check requirements such as required properties, value types and allowed fields. Select a schema that applies to the specific file or API version, and record where it came from. If the schema permits additional keys, an obsolete key may still pass. If it is stale, it can reject a valid new setting or accept a removed one.

The final gate uses the real consumer where practical: load the config with the documented tool, validate a manifest with the intended platform, or submit a request to a controlled test API. Prefer a documented validation or dry-run command when the tool provides one. Record whether the check used a local fixture, a test service or a live dependency. Acceptance of a request body also does not prove that the response shown in the docs matches what the service returns; compare the relevant response separately.

For API examples, connect body validation with the contract and endpoint checks described in testing API reference docs. For command examples that consume config files, see testing CLI docs.

Label fences for the format readers receive

A fence language is a small but useful contract. Use the label for the syntax actually shown, then describe any additional expectations in the surrounding text.

JSON has no comment syntax, and its grammar does not allow trailing commas. The JSON specification also says object names should be unique: duplicate names can produce unpredictable results across implementations.

JSONC and JSON5 are useful labels when they match the target consumer. The VS Code JSON documentation describes a JSON with Comments mode that accepts comments and permits trailing commas with a warning. The JSON5 specification permits additional forms, including comments, trailing commas and single-quoted strings. Neither label makes that content strict JSON.

For fragments, use prose immediately before the block: “The following is a fragment; add it inside the existing tool.acme_tool table,” for example. If the fragment is independently parseable, you can still check its syntax. If it relies on surrounding content or contains an ellipsis, record that the displayed block was not checked as a complete document.

Check JSON request and response bodies

Start with a parser for each complete JSON block. Save the exact content as request.json, then run:

python -m json.tool request.json

The command parses and pretty-prints the file; a syntax error produces a failure. It is useful for finding a missing comma, a comment or a trailing comma. See the Python JSON documentation for the command and parser behavior.

A successful result has an important limit: Python's default JSON parser accepts NaN, Infinity and -Infinity, and it keeps the last value for a repeated object name. Those behaviors are more permissive than the checks many teams want for interoperable JSON. The JSON specification forbids those numeric constants and recommends unique object names.

The following script uses only the Python standard library. Save it as check_json.py and run it with one or more file paths. It rejects those constants and duplicate keys at any object level.

import json
import sys
from pathlib import Path


def reject_constant(value):
    raise ValueError(f"Non-JSON numeric constant: {value}")


def unique_object(pairs):
    result = {}
    for key, value in pairs:
        if key in result:
            raise ValueError(f"Duplicate object key: {key!r}")
        result[key] = value
    return result


def main(paths):
    failed = False
    for name in paths:
        try:
            text = Path(name).read_text(encoding="utf-8")
            json.loads(
                text,
                parse_constant=reject_constant,
                object_pairs_hook=unique_object,
            )
        except (OSError, UnicodeError, ValueError) as exc:
            print(f"{name}: {exc}", file=sys.stderr)
            failed = True
        else:
            print(f"{name}: valid JSON")
    return 1 if failed else 0


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python check_json.py FILE [FILE ...]", file=sys.stderr)
        raise SystemExit(2)
    raise SystemExit(main(sys.argv[1:]))

For example:

python check_json.py request.json response.json

This is a stricter policy for duplicate keys than the specification's recommendation. It helps avoid a situation in which a reader's parser and the documented service choose different values. It still does not validate an API schema or prove that an endpoint accepts the body.

Test the contract around the body

A request example needs more than valid JSON. Check the method, path, required headers, authentication instructions and fields against the intended API version. A response example should match the promised status and shape, with variable fields identified as representative where appropriate.

Use safe fictional values in published examples. A value such as https://api.acme-docs-demo.test can communicate a URL's shape without suggesting a real service. If a test replaces a placeholder with a fixture, document the substitution and preserve a separate check of the published block's syntax.

Keep request and response examples distinct. A response is usually output to compare or inspect, not a request to send back to the service. If fields such as timestamps or generated identifiers vary, compare stable fields and document which values are illustrative.

Check TOML against the intended version

TOML appears in pyproject.toml and many tool configuration files. The parser can check the file's structure, while the tool must decide what its own tables and keys mean.

This is valid TOML 1.0.0:

[tool.acme_tool]
enabled = true
targets = ["docs", "guides"]
cache = { enabled = true, directory = ".cache/acme" }

With Python 3.11 or later, save the content as settings.toml and parse it using tomllib.loads:

python -c 'import pathlib, tomllib; tomllib.loads(pathlib.Path("settings.toml").read_text(encoding="utf-8")); print("TOML parsed")'

The Python 3.12 version of tomllib parses TOML 1.0.0. It reads TOML but does not write it. The Python 3.12 tomllib documentation describes both loads and the binary-file interface for load.

TOML 1.0.0 does not permit a key or table to be defined more than once. Its inline tables must remain on one line and cannot have a trailing comma. The following is deliberately broken for TOML 1.0.0:

cache = { enabled = true, directory = ".cache/acme", }

The TOML 1.0.0 specification defines those rules. The TOML 1.1.0 specification is a separate version; current Python tomllib documentation notes that Python 3.15 added TOML 1.1.0 support. A parser targeting 1.0.0 can therefore reject syntax intended for 1.1.0. Pin the parser used in CI to the compatibility target you want to promise.

After parsing, run the actual tool's documented config validation or a safe operation that loads the file. A valid [tool.acme_tool] table is still only an example until the relevant version of the tool accepts its keys. For pyproject.toml, also check that the example is placed under the table expected by the package or tool; TOML syntax alone cannot tell you that.

Check YAML structure and consumer behavior

YAML is common in CI workflows, application configuration and Kubernetes-style manifests. Indentation changes structure, and tab characters cannot be used for indentation. A document can also parse while expressing a different nesting from the one the author intended. The YAML 1.2.2 specification describes the indentation rules.

This is a complete, syntactically valid YAML document:

service:
  name: sample-api
  replicas: 2
  ports:
    - 8080

A YAML linter can catch parsing and style problems, including duplicate keys and indentation issues, according to its configured rules. Install yamllint and run it on a saved file:

python -m pip install --user yamllint
yamllint settings.yaml

The yamllint quickstart also describes checking directories and selecting output formats. Review the rule configuration used in CI: a style warning and an invalid document do not necessarily have the same significance, and a lint configuration determines which findings appear.

A linter cannot establish that a manifest is accepted by a particular cluster or that a CI platform supports every key. Add a schema check for the relevant version when available, then use the consumer's validation or dry-run facility in a suitable test environment. For a deployment manifest, distinguish a local structural check from a check against a real cluster and its installed APIs. For a CI workflow, distinguish valid YAML from a workflow accepted by the CI service.

Keep YAML fragments understandable

A short mapping can be valid YAML while still being only part of the file readers need. Say where it belongs and identify any required surrounding keys. If indentation must change when the fragment is inserted, show the containing structure or give a complete file alongside it.

Avoid using tabs to align nested examples. Preserve spaces when extracting fenced blocks for tests, and test the text as rendered in the source. A check that normalizes indentation before parsing can hide a copy-and-paste failure.

Treat look-alike formats as separate contracts

A file's appearance does not establish its parser. JSONC, JSON5, INI and environment-variable files can all look familiar while following different rules. Test each with the actual consumer or a parser configured for that consumer's dialect.

Do not claim that a JSON parser validated a JSONC or JSON5 example after stripping comments or rewriting its syntax. Such a transformation may be useful as part of a separate test, but the result describes the transformed content. The stronger check feeds the published content to the intended consumer and records any substitutions.

For an environment file, use non-secret placeholder values in the documentation and explain how readers supply their own credentials. A test can verify the loader's behavior with harmless fixtures, but that does not establish that a real credential works or that an external service is available.

Put the checks in CI

A practical workflow has two inputs: the documentation blocks readers see and the files or fixtures used by the checks. Keep them connected. If examples are copied into separate fixture files, compare them with the published blocks or generate one from the other so a passing fixture cannot drift away from the page.

The following GitHub Actions outline checks JSON and TOML fixtures and runs yamllint on YAML fixtures. It assumes the strict JSON script above is saved as check_json.py at the repository root and that the named files exist. Replace the paths with your own complete examples.

name: Check documentation data

on:
  pull_request:
  push:
  workflow_dispatch:

jobs:
  config-examples:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install YAML lint tool
        run: python -m pip install yamllint
      - name: Check JSON
        run: python check_json.py docs-fixtures/request.json docs-fixtures/response.json
      - name: Check TOML
        run: python -c 'import pathlib, tomllib; tomllib.loads(pathlib.Path("docs-fixtures/settings.toml").read_text(encoding="utf-8"))'
      - name: Check YAML
        run: yamllint docs-fixtures/settings.yaml

This job checks the named files, not every fenced block in the repository. Add a reliable extraction or synchronization step if the source of truth is Markdown. Preserve fence language, page and line information so a failure points reviewers back to the published example. Decide explicitly how your extractor handles tabs, indentation, alternate examples and intentional fragments.

Add schema and consumer checks as separate steps with clear names and pinned versions where appropriate. If a consumer check needs a service, credentials or a cluster, document its prerequisites and coverage. A skipped integration step should remain visible rather than being counted as evidence that the consumer accepted the example.

Run the fast checks on pull requests. Run version-sensitive acceptance checks against the release artifact before publication when they are intended to block a release, and repeat against the shipped version when practical. Scheduled runs help reveal changes in external dependencies. See the guides for GitHub Actions docs and GitLab CI docs for the surrounding CI examples themselves.

Triage failures in context

A failed check is a useful signal, but the right response depends on what the page promises and what the test actually did. Review the source block, nearby instructions, consumer version and failure message together.

A placeholder is not automatically harmless. If a page says “copy this complete file” and includes ..., the instruction and the example conflict. Either provide a copyable file or make the required edits explicit. Similarly, a syntactically valid placeholder value may still be rejected by the consumer; say when substitution is required.

Review failures that involve an external service separately from local syntax failures. A network outage, rate limit or missing test fixture can prevent an acceptance check from completing without proving that the documentation is wrong. Preserve the failure and its context so someone can reproduce it.

Where DocsAlive fits

DocsAlive extracts fenced blocks with their language, page and line, then plans whether to execute, syntax-check or skip them. It keeps supported executable examples in page order and reports findings with reasons and categories. This helps reviewers distinguish likely documentation bugs from context, credentials, fixtures and other causes of noise. The framework support reference describes supported documentation inputs.

Its structured-data behavior is specific:

Python's default JSON parser accepts NaN, Infinity and -Infinity and keeps the last value of a repeated key. Those blocks can therefore pass the product's JSON check despite the interoperability concerns described above. Use the stricter script or another suitable checker when those cases must fail.

For YAML, run yamllint or a YAML parser in your own CI. For every structured format, add schema and tool-version checks where needed: the engine does not validate a block against a schema or establish that a configuration key exists in the current version of a tool.

OpenAPI files in the repository receive a separate reference check: they are parsed as JSON, or as YAML when a YAML library is available, and local $ref values are resolved. This is not a schema validation of the documentation's fenced examples. The Growth plan lists OpenAPI reference checks; see pricing for plan details.

A local scan can help identify the blocks and their reported outcomes:

docsalive scan . --execute --out docs-report

The command requires the CLI to be installed. The output directory receives Markdown and JSON reports. Inspect the findings and reasons, including skipped blocks, before treating a summary as evidence of coverage. A health check is also available for a public repository you own or maintain: the team runs the engine on up to 40 documentation pages and emails the report within two business days.

A pasteable review checklist

Use this checklist in a pull request or adapt it to your documentation process:

• [ ] Identify each complete request, response, config file and intentional fragment.
• [ ] Label fences for the syntax shown: json, jsonc, json5, toml, yaml, yml, ini, dotenv or env as appropriate.
• [ ] Make copyable examples complete, and explain required substitutions.
• [ ] Mark omitted content and say where each fragment belongs.
• [ ] Parse complete JSON examples and reject non-finite constants and duplicate keys if your policy requires it.
• [ ] Parse TOML with the format version readers are expected to use.
• [ ] Lint or parse YAML, including indentation and duplicate-key checks.
• [ ] Test JSONC, JSON5, INI and environment files with their intended consumers.
• [ ] Validate against the applicable schema and record its version or source.
• [ ] Check the example with the intended tool or API version where practical.
• [ ] Compare documented responses or results with stable observed fields.
• [ ] Keep test fixtures synchronized with the published blocks.
• [ ] Record skipped checks, required credentials, fixtures and external dependencies.
• [ ] Review failures in context and fix examples that break their stated promise.
• [ ] Run relevant checks on documentation changes, releases and a schedule appropriate to dependency drift.

The broader documentation testing checklist can help connect these checks with installation, executable examples and output. For options to combine in a local workflow, see free docs testing tools.

The sources we checked

These sources were read on November 10, 2026.

FAQ