Testing docs examples that need API keys

The short answer

Split credential-dependent examples into testable layers: check syntax, run request construction and response parsing offline, and run the live call separately against an authorized test-mode or sandbox account. Use an environment variable with a clear name, document how readers obtain and provide their own value, and report a skipped live call as a coverage gap.

Credential-gated examples are among the hardest documentation blocks to test because a valid run depends on access that a clean scanner should not assume. They are also a common reason a scan cannot run a block. A missing key does not by itself mean the documentation is broken; a reader who cannot work out where to put their key has a documentation problem.

DocsAlive detects certain secret reads and placeholders, classifies credential-dependent blocks, and does not run them with customer-provided test credentials. Python and JavaScript blocks in that category still receive syntax checks; other affected blocks are skipped. Run the live leg in your own authorized CI. See documentation code testing and SDK documentation testing for the broader workflow.

Why credentials complicate documentation tests

An API example often combines several promises in a few lines: the package can be installed, the client can be configured, the request has the right shape, the key is accepted, the account has access, and the response can be parsed. A failed live call does not immediately identify which promise failed.

A clean documentation check may intentionally have no access to a key. A forked pull request may also run without repository secrets. A sandbox account can be unavailable or lack a permission needed by the example. Those conditions should be visible in the result, but they do not automatically establish that the code shown to readers is wrong.

At the same time, treating every authentication failure as harmless can hide a real documentation defect. A page may use a placeholder without explaining which environment variable receives the real value. It may point readers to the wrong kind of credential, use an obsolete header, or omit a required configuration step. Triage must distinguish an unavailable test credential from instructions that a reader cannot follow.

The best coverage comes from separating the promises. Syntax and request construction can often be checked without a live service. A recorded response can exercise parsing. A live test can then focus on authentication and current service behavior under a controlled account. Each result says what was tested and what remains unknown.

Make placeholders honest and recognizable

Use a named environment variable for a value that varies between environments. The following complete Python program reads a fictional Acme SDK credential without embedding a key:

import os

api_key = os.environ.get("ACME_API_KEY")
if api_key:
    print("A key is configured")
else:
    print("Set ACME_API_KEY to run the live example")

ACME_API_KEY and the Acme SDK are fictional names in this guide. The program reports only whether a value is present; it never prints the value. Python's environment documentation describes os.environ and os.getenv. The Twelve-Factor configuration guidance explains the broader practice of keeping deployment-specific configuration in the environment.

If a page must show a placeholder, use an unmistakable marker such as YOUR_API_KEY or <API_KEY>. Explain exactly which value replaces it and where the reader supplies that value. Do not paste a fake key designed to look real: it can be mistaken for an actual credential and is harder for a reader or scanner to distinguish from one.

A placeholder is not a substitute for a configuration guide. State whether the value is a test-mode key, which environment variable the example reads, and which account or permission the live call requires. Avoid promising that a particular credential will work in every environment.

Separate the parts that need a key

A useful example has a clear boundary between local work and the external call. Depending on the SDK, client construction can be a local operation even when the client eventually needs credentials. Request building and parsing a recorded response can usually be tested without contacting the service. If a particular SDK validates or contacts the service during construction, treat that operation as part of the live boundary instead.

Design the example so the offline layer is meaningful. Build the same request shape used by the live client, return a representative response, and assert the fields the documentation promises. Keep recorded fixtures free of real credentials and private customer data. When a service response changes, review whether the fixture and documentation should change together.

A live test remains valuable for behavior a stub cannot establish: authentication, permissions, current endpoint behavior, and the interaction with the actual SDK or service. Use a test-mode or sandbox account that the owner has authorized for CI. Give it only the access the test needs and make any cleanup or service-side effects part of the test design.

Run an offline Python example

The following single file uses only the Python 3.12 standard library. It starts a local HTTP stub in a background thread, sends a request, parses a canned JSON response, checks the result, and shuts the server down. The host is local; no external API is contacted.

Save it as offline_example.py:

import json
import os
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.request import Request, urlopen


class StubHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/v1/greeting":
            self.send_error(404)
            return

        body = json.dumps({"message": "Hello from the stub"}).encode("utf-8")
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        pass


def fetch_greeting(base_url):
    api_key = os.environ.get("ACME_API_KEY", "offline-placeholder")
    request = Request(
        base_url + "/v1/greeting",
        headers={"Authorization": "Bearer " + api_key},
    )
    with urlopen(request, timeout=5) as response:
        return json.load(response)


def main():
    server = HTTPServer(("127.0.0.1", 0), StubHandler)
    thread = threading.Thread(target=server.serve_forever, daemon=True)
    thread.start()
    try:
        host, port = server.server_address
        result = fetch_greeting(f"http://{host}:{port}")
        assert result["message"] == "Hello from the stub"
        print(result["message"])
    finally:
        server.shutdown()
        thread.join()
        server.server_close()


if __name__ == "__main__":
    main()

Run it with:

python3 offline_example.py

It prints:

Hello from the stub

The default value is explicitly an offline placeholder. The stub does not validate it, and the example does not prove that an actual service accepts any credential. The client function reads ACME_API_KEY so the same configuration boundary is visible, while the local server makes the request and parsing path runnable without a key.

For a real SDK, avoid sending an actual credential to a stub unless the test is deliberately designed and authorized to do so. The stub can verify a harmless test marker or inspect request structure without logging sensitive headers. Keep the live endpoint and live credential in a separate test configuration.

A recorded fixture is another offline option. Store a representative, sanitized response in the repository, load it in a test, and assert the parsing result. A fixture is easier to maintain than a server for pure parsing, while a local stub can also exercise HTTP request construction and status handling. Neither replaces a live authentication check.

Skip a live test cleanly when the key is absent

The following shell script checks for the fictional environment variable and exits successfully with a clear skip message when it is absent. It does not make a network request or print the key.

if [ -z "${ACME_API_KEY:-}" ]; then
  printf '%s\n' 'SKIP: ACME_API_KEY is not set'
  exit 0
fi

printf '%s\n' 'Credential is present; run the authorized live test here'

Save it as a script and run it in the environment where the live test would execute. Replace the final message with your project's actual live-test command after configuring an authorized sandbox. Keep the skip visible in CI output and reports: a successful job that skipped the live call is not evidence that the call passed.

For a pytest suite, the following complete test file uses pytest as its only non-standard-library import. It demonstrates the skip condition without making a real API call. Save it as test_live_example.py:

import os

import pytest


@pytest.mark.skipif(
    not os.environ.get("ACME_API_KEY"),
    reason="ACME_API_KEY is required for the authorized live test",
)
def test_live_configuration_is_present():
    assert os.environ.get("ACME_API_KEY")

Run it with pytest test_live_example.py in an environment where pytest is installed. When the variable is absent, pytest reports a skip. When it is present, this demonstration checks only that the value exists; replace the test body with your authorized sandbox call and assertions to verify live behavior. The pytest skip documentation describes skipif and its reason.

Do not print the value in a failure message, command trace, HTTP log, or test artifact. A test can report that a credential is missing or rejected without recording the credential itself.

Configure CI around the trust boundary

Store the sandbox credential in the CI platform's secret store and map it to a job-level environment variable for the live job. In GitHub Actions, secrets cannot be referenced directly in an if: conditional. Set the secret as a job-level environment variable, then check the environment variable in the step condition, as described in GitHub's secrets documentation.

A practical workflow has an offline job that runs for ordinary changes and a live job or step that runs only when the authorized secret is available. The live step should report a skip when it cannot run. Map the secret in the workflow using GitHub's documented expression syntax; the exact mapping belongs in your repository's workflow configuration.

With the exception documented by GitHub for GITHUB_TOKEN, secrets are not passed to runners for workflows triggered from forks. Design the live job so it is not a required check on fork pull requests. Keep syntax, offline tests, and other safe checks available there, and run the live leg later in an authorized context. Do not expose a secret to untrusted pull request code merely to make a check green.

Mask sensitive values in logs, including non-secret sensitive values where appropriate, and never deliberately print keys. GitHub documents the ::add-mask::VALUE mechanism for masking a value; arrange masking without echoing the underlying value into an unprotected log. Review shell tracing, HTTP debugging, exception messages, and uploaded artifacts as well as the normal test output.

A useful result should make the coverage distinction explicit:

• Offline checks passed and the live test passed.
• Offline checks passed and the live test was skipped because no credential was available.
• Offline checks passed and the live test failed with an authentication or service response.
• Syntax failed before either behavioral layer could run.

Those outcomes call for different reviews. A skipped live test is an honest result, but a release process that requires live verification must arrange an authorized run before treating that requirement as met.

Triage credential-related findings

Record the test layer, endpoint type, account type, and sanitized error. Do not include the key in the finding. Use the evidence to decide whether the issue is in the page, the test environment, or an external service.

needs-credentials is not itself a docs-bug classification. It says the scan cannot establish the live result without credentials. A separate human review can still find a documentation problem, such as a placeholder that readers cannot map to the configuration they are asked to supply.

A 401 or 403 also needs context. It can result from a missing or invalid key, insufficient account permissions, or a setup mismatch. Preserve enough sanitized evidence to reproduce the problem with an authorized account, then fix the instructions if they led the reader to the wrong configuration.

Where DocsAlive fits

Before running a block, DocsAlive looks for secret environment reads in supported Python and JavaScript forms. It identifies variable names containing KEY, TOKEN, SECRET, PASSWORD, or CREDENTIAL, including Python environment reads and JavaScript process.env reads. It also recognizes placeholder patterns such as YOUR_API_KEY, <API_KEY>, and relevant shell variables.

When a block has a secret or placeholder requirement, Python and JavaScript are syntax-checked only; other blocks, including shell and curl examples, are skipped. The finding is classified as needs-credentials, which is not counted as a docs bug. For example, a curl block using $ACME_API_KEY is skipped rather than sent. A Python syntax error can still be caught, but a successful syntax check does not test the API call.

Triage also recognizes authentication-related output, including 401 Unauthorized, 403 Forbidden, and invalid or missing API key messages, as needs-credentials. These classifications help distinguish access-dependent results from likely documentation bugs, but they do not replace a review of whether the instructions are understandable.

The sandbox has no access to customer secrets, and running examples with customer-provided test credentials is not currently supported. Run the live leg in your own authorized CI with a test-mode or sandbox credential. Use the product's report to see what was syntax-checked or skipped, and combine it with the offline and live results from your own tests. See findings for the reporting model, and testing AI and LLM SDK documentation examples for provider-specific credential cases.

Checklist

Copy this list into a pull request or release checklist:

• [ ] Name each required credential and explain where the reader supplies it.
• [ ] Use an environment variable or an unmistakable placeholder, never a real-looking pasted key.
• [ ] Separate local construction, request building, parsing, and the live call where practical.
• [ ] Run syntax checks even when no credential is available.
• [ ] Test request and response handling with a local stub or sanitized fixture.
• [ ] Use an authorized test-mode or sandbox account for live verification.
• [ ] Store the credential in the CI secret store and map it to a job-level environment variable.
• [ ] Skip the live step clearly when the secret is absent.
• [ ] Keep the live job from being required on fork pull requests.
• [ ] Mask sensitive values and check logs and artifacts for accidental disclosure.
• [ ] Record which layer passed, failed, or was skipped.
• [ ] Review unclear placeholders as documentation issues.

The sources we checked

These sources were read on November 10, 2026.

FAQ