Testing Java documentation examples

The short answer

Test Java documentation examples according to what they promise: run complete programs, compile source files with the JDK, check fragments in the context readers are given, and verify stable output with assertions. Test build instructions against the stated JDK and dependency versions. Record examples that need a larger project, a service, or credentials so a skipped check does not look like a successful run.

Java examples appear in READMEs, SDK guides, and API reference pages, but those places make different assumptions. A README may show a complete first program. An SDK guide may assume a project and dependency already exist. Javadoc may show just the statement that matters. The test should reflect those assumptions and make them visible.

DocsAlive recognizes Java fences but does not compile or run them. It reports them as skipped with the reason not executable (java). Use your own JDK checks for Java, while using the product to check supported blocks on the same page. See documentation code testing for the broader workflow and SDK documentation testing for release-specific checks. PHP pages follow the same pattern; see testing PHP documentation examples.

Why Java documentation examples break

Java source is explicit about the environment around an expression. A line that looks complete in a page may rely on an import, a class declaration, a method, an exception handler, or a dependency that the reader has not been told to add. Those dependencies are easy to overlook when an author copies code out of a working project.

API changes add another source of drift. A class or method can be renamed between library releases while the documentation retains the old spelling. A snippet may compile against a local checkout and fail against the artifact a reader installs. Maven or Gradle coordinates can point to the wrong group, artifact, or version even when the Java source itself is correct.

The JDK version matters too. A page that uses a language feature introduced in a later release needs to state its minimum JDK. A maintainer running a newer JDK can otherwise miss a failure experienced by a reader following the published prerequisites. Test the stated minimum as well as any newer versions you explicitly support.

Checked exceptions are a particularly useful example of context. A method call can be valid in an enclosing method that declares throws, while the same call fails when copied into a main method without that declaration or a handler. A fragment can be accurate, but its surrounding instructions must tell the reader where it belongs.

Finally, output can drift independently of compilation. A program may still run while printing a result different from the one shown on the page. Compilation checks the source; an output assertion checks a separate promise.

Classify each block before running it

A fence label tells you the language, but it does not establish that the contents form a complete program. Record the intended unit and the context the page supplies.

A complete program should identify its filename when the filename matters. A fragment should say what surrounds it and which imports or prior steps it needs. Avoid silently supplying a test wrapper that gives the example an import or exception handler the reader never receives.

For example, a fragment intended for JShell can use a declaration followed by an expression. It does not need a class or main, but a reader should not be told to save that fragment as a standalone program. Conversely, a class with a main method should be checked as a program rather than accepted solely because its individual statements work interactively.

Build examples deserve a separate check. A pom.xml or Gradle configuration is not Java source, and compiling a standard-library example cannot establish that its coordinates resolve. State the repository, artifact, version, and JDK needed for the actual project. Test the published artifact when the page promises an installation path, and distinguish that result from a check against a local checkout.

Run a complete program with the JDK

Save the following complete program as Example.java. It uses only the Java standard library and runs on JDK 21.

public class Example {
    public static void main(String[] args) {
        String message = "Documentation examples should run";
        System.out.println(message);
    }
}

Run it with the source launcher:

java Example.java

The expected output is:

Documentation examples should run

The source launcher was delivered in JDK 11 by JEP 330. For a single source file, it compiles in memory and launches the first top-level class's main method; it does not write the compiled classes to disk. This is useful for short, dependency-free examples because a reader can follow the page without first creating a build project.

That convenience has a boundary. Multi-file source-code launching arrived in JDK 22 under JEP 458. Do not describe a multi-file source-launcher command as a JDK 21 feature. On JDK 21, compile related source files explicitly or use the build process the project documents. If a program needs third-party libraries, provide and test the necessary classpath or build instructions.

Compile with warnings visible

Compilation is a useful independent check even when you also run the program. The following commands compile the file into a temporary directory and remove that directory when the shell exits:

work_dir="$(mktemp -d)"
trap 'rm -rf "$work_dir"' EXIT
javac -Xlint:all -Werror -d "$work_dir" Example.java
java -cp "$work_dir" Example

-Xlint:all requests compiler warnings, and -Werror makes warnings fail compilation. This is a deliberately strict policy: decide whether your documentation examples should meet it, and review a warning in the context of the JDK you support. A warning does not automatically mean the page gives a reader unusable instructions.

Compiling into a temporary directory keeps generated class files out of the documentation tree. It also makes the files and classpath used by the check easier to see. For a real project, include all necessary source files and dependencies, and ensure the test does not accidentally inherit a classpath from the maintainer's environment.

When the page promises compatibility with a minimum JDK, run this check on that JDK. A successful compile on JDK 21 alone does not establish compatibility with an earlier release. Likewise, a source-launcher check does not establish that Maven or Gradle coordinates are correct.

Turn a promised result into an assertion

A program that exits successfully may still produce the wrong value. Save this separate complete program as OutputCheck.java:

public class OutputCheck {
    public static void main(String[] args) {
        String actual = "hello".toUpperCase(java.util.Locale.ROOT);
        assert actual.equals("HELLO") : "Unexpected result: " + actual;
        System.out.println(actual);
    }
}

Run it with assertions enabled:

java -ea OutputCheck.java

The expected output is HELLO. If the asserted value changes, the process fails with an assertion error. The -ea flag matters: Java assertions are disabled unless enabled. A CI command that omits the flag can report success without checking the assertion.

Use assertions for stable facts that the page promises. If output includes a timestamp, a generated identifier, or a service-controlled field, check the invariant that matters and explain which parts can vary. Do not normalize away a difference merely to make a test pass; first decide whether the difference changes the reader's understanding.

Test fragments with JShell

JShell is a REPL introduced in JDK 9. It evaluates declarations, statements, and expressions without requiring a surrounding class and main method. The JDK 21 JShell guide describes the tool and its interactive model.

For a repeatable check, save this complete script as fragment.jsh:

import java.util.List;
var names = List.of("Ada", "Lin");
System.out.println(names.size());
/exit

Run it with:

jshell --feedback concise fragment.jsh

The script prints 2, along with any JShell feedback produced by the selected mode. The /exit command ends the session. When comparing output, account for JShell's own feedback rather than assuming its whole transcript is identical to output from a standalone Java program.

JShell is useful when the published fragment is genuinely intended to be entered in a REPL. It is not a universal wrapper for source fragments. A statement copied into a method, a declaration inside a class, and an expression entered at a JShell prompt have different contexts. Test a fragment in the context the page instructs readers to use.

If an example depends on an import, put the import in the published path or explicitly identify the earlier step that supplies it. If it depends on a checked exception, test the enclosing method and its exception handling. If it relies on an SDK, provide the relevant dependency to the test rather than assuming JShell contains it.

A normal project test suite is also a useful home for examples that share production setup or need stronger assertions. The JUnit user guide describes organizing tests. Keep the documented source and the tested source connected so a test cannot pass after the page has diverged.

Keep Javadoc snippets testable

JDK 18 introduced the Javadoc @snippet tag through JEP 413. An external snippet can point to a named region in a source file. Javadoc looks for external files under a snippet-files subdirectory of the package source directory, or on a path supplied with --snippet-path.

A Javadoc comment can refer to a region like this:

/**
 * Shows a small calculation.
 * {@snippet file="Calculation.java" region="example"}
 */
public class CalculationDocs {
}

The following is a separate file named Calculation.java. Its marked region is suitable for inclusion, and the whole file can be compiled and run:

public class Calculation {
    public static void main(String[] args) {
        // @start region="example"
        int answer = 6 * 7;
        System.out.println(answer);
        // @end
    }
}

Place the external file where the Javadoc configuration can find it, then compile and run that file as part of your checks. Verify the rendered documentation as well: a source file can compile while the selected region omits a line the reader needs. The Javadoc snippets guide covers region markup and presentation.

Inline snippets are convenient for short illustrations, but Javadoc itself does not validate their code. If an inline snippet is meant to be copied, arrange a separate compile or test for the corresponding source. For external files, compilation checks the file as a whole; it does not automatically prove that every isolated region is complete in the context where it is displayed.

Extract Java fences for a first-pass check

The following Python 3 standard-library script extracts simple fenced java blocks from a Markdown file. It expects each selected block to be a complete, dependency-free Java source file with a public class. It saves each block under that class's filename, compiles it with strict warnings, and runs it. It reports blocks that do not meet those assumptions for manual classification.

Save it as check_java_fences.py:

import pathlib
import re
import subprocess
import sys
import tempfile

if len(sys.argv) != 2:
    raise SystemExit("Usage: python check_java_fences.py PAGE.md")

page = pathlib.Path(sys.argv[1])
text = page.read_text(encoding="utf-8")
blocks = re.findall(
    r"(?m)^```java[ \t]*\r?\n(.*?)^```[ \t]*$",
    text,
    flags=re.DOTALL,
)
failed = False

with tempfile.TemporaryDirectory() as temporary:
    root = pathlib.Path(temporary)
    for number, source in enumerate(blocks, start=1):
        match = re.search(r"\bpublic\s+class\s+([A-Za-z_]\w*)", source)
        if not match:
            print(f"Block {number}: review as a fragment or other source")
            continue
        name = match.group(1)
        block_dir = root / str(number)
        block_dir.mkdir()
        source_file = block_dir / f"{name}.java"
        source_file.write_text(source, encoding="utf-8")
        compile_result = subprocess.run(
            ["javac", "-Xlint:all", "-Werror", "-d", str(block_dir),
             str(source_file)],
            check=False,
        )
        if compile_result.returncode:
            print(f"Block {number}: compilation failed")
            failed = True
            continue
        run_result = subprocess.run(
            ["java", "-ea", "-cp", str(block_dir), name],
            check=False,
        )
        if run_result.returncode:
            print(f"Block {number}: execution failed")
            failed = True
        else:
            print(f"Block {number}: compiled and ran")

if not blocks:
    print("No simple java fences found")
if failed:
    raise SystemExit(1)

Run it from a shell with Python and a JDK on PATH:

python3 check_java_fences.py docs/page.md

This is a starting point, not a complete Markdown parser. It handles unindented triple-backtick fences labeled exactly java; it does not understand nested fences, attributes, tab groups, or page-specific setup. It also assumes the public class has a runnable main, has no package declaration requiring special handling, and needs no external dependencies. A Javadoc declaration or a deliberately broken example can therefore fail this first pass even when the page labels it correctly.

Review untested fragments and zero-block results. Extend the check with an explicit manifest for filenames, dependencies, expected output, and intentional failures as your documentation grows. A clean compile of extracted text alone cannot establish that surrounding installation instructions work.

Put the checks in CI

A useful CI path runs the examples when documentation or public APIs change, against the minimum supported JDK and any additional versions your project commits to support. The following plain GitHub Actions outline checks out a repository, installs JDK 21, and runs the extraction script. It assumes the script and page exist at the shown paths.

name: Java documentation examples

on:
  pull_request:
  push:
  workflow_dispatch:

jobs:
  java-docs:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: '21'
      - name: Check complete Java fences
        run: python3 check_java_fences.py docs/page.md

Treat this as an outline for the simple blocks the script supports. Add separate commands for JShell scripts, Javadoc generation, project-based examples, and output checks. If a page shows a Maven or Gradle dependency, run its documented build in a clean project with the intended repository access; a JDK-only fence check cannot validate that dependency.

On a release, test the artifact intended for readers. A pre-release check can block publication if that is your policy; a later check against the published artifact verifies the actual installation path but cannot retroactively prevent publication. Scheduled runs can find drift in dependencies between releases. See GitHub Actions documentation testing for the wider CI pattern.

Triage the result in context

A failing command is evidence to investigate, not a complete diagnosis. Keep the source, JDK version, dependency versions, command, and relevant output with the finding.

A missing setup instruction can be a reader-facing problem even when the code works in the author's project. On the other hand, a fragment labeled as such is not necessarily defective because it lacks a class. Preserve that distinction in the report and make the page's intended copy-and-paste boundary explicit.

Where DocsAlive fits

DocsAlive extracts fenced blocks and reports what it plans to execute, syntax-check, or skip. Java fences are recognized and skipped with the reason not executable (java): they are not compiled, linted, or run. The supported stacks page lists Java among the languages that are recognized but not executed yet. A Java page therefore needs its own javac, java, JShell, or project-test checks for Java coverage.

The rest of the page can still be checked according to the supported language handling. Shell blocks and curl requests can run, while JSON and TOML blocks receive syntax checks. A shell block invoking java in the default runner image would be reported as missing-tool; the default image is not a JDK image. A skipped Java block remains visible in the report rather than becoming evidence of a passing Java test.

Use the report alongside your JDK results. The two answer different questions: one exposes the product's coverage of the page and its skips, and the other establishes what your Java commands actually compiled and ran. See findings and classifications for the report categories. To see how a scan reports a page that mixes Java with shell and JSON blocks, you can request a health check for a public repository you own or maintain; the team runs the engine on up to 40 of its docs pages and emails the report. For a compiled-language comparison, see testing Go documentation examples and testing Rust documentation examples.

Checklist

Copy this list into a pull request or release checklist:

• [ ] Classify each Java block as a complete program, fragment, Javadoc snippet, or build configuration.
• [ ] State the minimum JDK and test on that version.
• [ ] Check imports, class names, method names, and checked exceptions in the reader's context.
• [ ] Compile complete source with the intended dependencies and review warnings.
• [ ] Run complete programs and assert stable promised results with assertions enabled where used.
• [ ] Test JShell fragments as scripts or test fragments in their documented enclosing source.
• [ ] Resolve documented Maven or Gradle coordinates in a clean project.
• [ ] Compile external Javadoc snippet files and inspect their rendered regions.
• [ ] Record skipped blocks, prerequisites, fixtures, and live-service coverage.
• [ ] Repeat relevant checks for public API changes and release artifacts.

The sources we checked

These sources were read on November 10, 2026.

FAQ