On this page
The short answerWhy installation instructions break firstDefine the reader's starting pointTest pip commands as writtenTest npm install and npm ci for their intended pathsUse a clean container for the whole pathPair installation with the first useConfigure and interpret a product runTriage installation failuresInstallation testing checklistWhat this does not replaceThe sources we checkedFAQThe short answer
Test installation instructions as a reader follows them: start with the stated environment, run the exact documented install command, and then run the first import or command the page promises. Keep those steps together in a clean environment. Record the package source, runtime, lockfile and any prerequisites supplied outside the page, so a passing result has a clear meaning.
Installation can fail before a reader reaches the first example. A package name, extra, lockfile, runtime or system dependency may no longer match the instructions. Test the relevant path when the docs or package changes, when a release is prepared, and periodically for dependency drift. The broader documentation testing guide covers how this fits with the rest of a documentation test suite.
DocsAlive infers install steps documented on a page and runs supported examples in page order in a fresh container per page. It reports a documented pip or npm package that does not exist as package-not-found, and reports a missing install step as a hint rather than automatically filing it as a docs bug. It has limits: credential-dependent examples are classified rather than executed, and an install test does not prove every platform or installer works.
Why installation instructions break first
An install section is a chain of assumptions. The package must be available under the name shown, the reader must have the necessary runtime and tools, the command must select the intended dependency set, and the next example must use what was installed. If the first link breaks, a reader cannot reach the rest of the guide even if every later code block is correct.
A package can be renamed while an old page keeps the previous name. An optional feature may require an extra that the basic install command does not request. A repository may change its dependency declarations without updating the lockfile that a clean install expects. The page may assume a Python or Node environment different from the one a reader starts with, or it may omit a system package needed in the intended environment. These are different problems and call for different evidence.
Make a distinction between a package failure and an environment failure. If the name in the page is wrong, change the instruction after verifying the intended distribution. If the name is right but a network or registry service cannot be reached, investigate that external condition before rewriting the page. If a system tool is absent, determine whether the page promised that it was already available or should have explained the prerequisite.
The quickstart guide shows how installation fits into an ordered onboarding journey. The SDK guide covers connecting examples to the artifact readers receive. For command-line products, the CLI docs testing guide focuses on the first invocation and its output.
Define the reader's starting point
Before automating an install, write down the environment the page addresses. Identify the operating system or image, the runtime, the package source, any required system tools, the current directory and the files the reader is expected to have. Identify the exact command a reader copies, including extras, flags and file paths. If the page offers alternatives, record each as a separate path.
This inventory gives a failure a useful context. A command that expects a checkout should be tested from a checkout; a command that installs a published package should be tested against that distribution when practical. A test that installs directly from a local working tree can be valuable for a change under review, but it answers a different question from a test of the published package name. Record both when both are relevant.
A clean environment is important because an existing package can make an incomplete page appear to work. So can a leftover lockfile, a previously generated file or a tool installed outside the documented flow. Start again for each independent path. Within the path, retain the state that the reader creates, including the installation, before running the first example.
Do not add hidden repairs to the test. If the harness must provide a prerequisite, record it and compare it with what the page says. A page may reasonably assume a prepared environment, but it should give readers enough information to recognize that assumption. If a test installs the product before a page that never mentions installation, the passing result alone cannot resolve whether the page is complete.
Test pip commands as written
The pip install reference describes several forms of installation input. A command can name a requirement specifier, install from a requirements file using -r, or request an editable install with -e. Extras and version constraints can change the requested dependencies; the Python Packaging User Guide discusses installing packages and optional dependencies. Test the form the page actually gives readers, including its files and current directory.
For example, the following commands illustrate two different documented paths using a fictional package. They are examples of command syntax, not claims that this package exists:
python -m pip install acme-docs-demo
python -m pip install 'acme-docs-demo[extra]'
The second command requests an extra named extra; a real guide should name an extra the real distribution provides and explain when it is needed. Do not treat a successful basic install as evidence that an example requiring an extra will work. Test the extra and the first feature that relies on it together. If the docs specify a version range, preserve that requirement in the test rather than silently replacing it with an unconstrained install.
A requirements file adds another source of instructions. Test the documented -r command with the actual file available at the stated path, and review changes to that file as changes to the reader's installation path. The pip reference also documents -c for a constraints file. If a page tells readers to use one, include that file and option in the test and record them with the result. Do not substitute a different file simply because it is convenient in CI.
An editable install is a separate journey. The -e form is relevant when a page directs a reader to work from a local project. Make sure the test begins with the checkout and directory the instructions assume, then run the command and the first documented use. A successful editable install from a checkout is not evidence that the published package can be installed by name. Conversely, a package install by name does not verify the checkout instructions.
For Python-specific examples, the Python docs guide covers imports, execution and expected output. If the installation instructions live in a repository introduction, test the README path as well as any longer guide that repeats or modifies it.
Start with a virtual environment
The Python standard library's venv documentation describes creating a virtual environment. It has its own package installation directory and is isolated from packages in the base environment by default. For a local test on a system with a suitable Python interpreter, create one and invoke its Python directly:
python -m venv /tmp/acme-docs-test-venv
/tmp/acme-docs-test-venv/bin/python -m pip install acme-docs-demo
/tmp/acme-docs-test-venv/bin/python -c 'import acme_docs_demo'
The package and import names in this example are fictional. Replace them with the exact names promised by the real page; a distribution name and the name used in an import need not look identical. The example uses a Unix-style environment path. Use the appropriate path for the platform being tested and start from a new environment for a fresh result.
A virtual environment isolates Python packages; it does not recreate an entire operating system or supply missing system tools. Record those prerequisites separately. If the guide supports several Python environments, repeat the reader path in the ones you claim to support. A single successful run establishes a result for its particular environment and inputs.
Test npm install and npm ci for their intended paths
A reader adding a package may be instructed to use npm install. A repository setup may instead instruct a reader to use npm ci. Follow the page's command rather than treating the two as interchangeable. The npm install reference describes installation using package specifications. For example, this fictional scoped package illustrates the form of an install command:
npm install @acme/docs-demo
The name is illustrative; use a real distribution and the real page's first command or import in an actual test. Keep the working directory and project files consistent with the page's instructions. If a page tells a reader to add a dependency to an existing project, test that journey in a suitable project rather than assuming a repository's own setup represents it.
The npm ci reference says that npm ci requires an existing package-lock.json or npm-shrinkwrap.json. If the lockfile and package.json dependencies do not match, it exits with an error instead of updating the lockfile. It installs an entire project, removes an existing node_modules before installation, and does not write to package.json or the lockfiles. It cannot add an individual dependency with that command.
Those details make lockfile consistency part of the documented repository setup. Test the committed project files together. If a dependency declaration changes, verify that the lockfile presented to readers is consistent with it. The npm reference also notes that flags affecting the shape of the dependency tree when the lockfile was created need to be supplied to npm ci as well. Record relevant project configuration rather than adding an unexplained flag only in the test.
A successful npm ci demonstrates the tested project install under its stated inputs. It does not establish that a separate npm install @acme/docs-demo instruction works for a new consumer, or that the first command in the documentation works. Test those reader paths separately. The JavaScript docs testing guide explains how to assess subsequent examples and their runtime assumptions.
Use a clean container for the whole path
A container can make the environment and the boundary of a test explicit. The Docker run reference says docker run starts a command in a new container, pulling the image if needed, and --rm removes the container and associated anonymous volumes when it exits. Choose an image that actually contains the tools needed for the path and record its identity.
The following is an illustrative shell outline. It assumes the named local image contains Python and pip, and that the fictional package and import are replaced for a real test. The install and import run in the same container:
docker run --rm acme-docs-test-image sh -c \
'python -m pip install acme-docs-demo && python -c "import acme_docs_demo"'
This creates a useful boundary, but the result still depends on the image, package source and network available during the run. A prebuilt image may contain tools or packages a reader does not have. Record those facts; if the image supplies a prerequisite, confirm that the documentation also tells the reader how to obtain it or explicitly assumes it. Use an image appropriate to each supported environment rather than presenting one container as proof for all platforms.
DocsAlive uses a fresh container per page, preserving page order for supported blocks. The default runner image contains Python 3.12 and Node 20. Use --image to select another runner image when the documented environment needs a different setup, and record which image produced the result. Each page begins separately, so a page cannot rely on a previous page's container having installed a package. A multi-page journey needs its own coverage in addition to page-level results.
The product's containers run as an unprivileged user with a read-only root filesystem, dropped Linux capabilities, no privilege escalation, and resource and time limits. Those conditions matter when an instruction expects host-level changes. A failure under those conditions should be understood in context; it is not automatic proof that a command fails on every reader's machine. See how it works for the pipeline and the CLI reference for options.
Pair installation with the first use
An installer can exit successfully while the next step still fails. The package may not expose the import or command the page shows, the example may depend on an optional feature, or the instructions may point at a different artifact from the one tested. Run the install and the first meaningful use in the same fresh environment, in the same order as the page.
For a Python library, that first use might be an import and a small operation. For a command-line tool, it might be the first invocation and a stable result the documentation promises. For a JavaScript package, use the import or command actually shown on the page. The aim is to verify the connection between the installed artifact and the next instruction, not merely that a package manager returned successfully.
Compare meaningful output when the page promises it. If output contains variable values, identify the stable part that matters and review the rest in context. A changed output can be a stale example, a legitimate variation or a sign that the wrong version or artifact was installed. Retain the command, output, environment and expected result so a maintainer can reproduce the distinction.
A quickstart may then require a credential, a local file or an external service. Record where the automated path stops. Do not insert real credentials into a generic example test. A successful install and import are useful evidence, but they do not verify an authenticated call that was not run. The quickstart testing guide helps map that larger journey.
Configure and interpret a product run
DocsAlive infers install steps that the page documents, then plans supported blocks in page order. A scan without --execute extracts and plans; add --execute to run the plan. For a local checkout, a focused example is:
docsalive scan . --execute --include 'docs/install*' --out docs-test-report
Adjust the include pattern to match the real paths. Review report.md and report.json in the output directory for the status, category and reason, including skips and hints. A pass may represent execution or a syntax check; examine the treatment before claiming that an entire installation journey was executed.
Use repeatable --setup CMD for a shell command that must run at the start of each page's container. For example, a repository may deliberately test its local package with --setup 'pip install -q .'. Make clear that this is test setup, and compare it with the page's own install instructions. Supplying the install in setup can make a later example work even when the page does not tell the reader to install the package.
Use --image to select an appropriate image. --no-network runs containers without network access, which can be useful when the test is meant to use dependencies already available in that environment. An instruction that needs a network package source may fail under that setting; interpret the result against the intended reader path. Do not call an offline failure a package-name error without evidence.
The package-not-found docs-bug category identifies a documented pip or npm package name that does not exist. missing-install-step is reported as a hint, not automatically filed as a docs bug: the page may assume a prerequisite, or it may need an instruction. Investigate what the page promises and what the test supplied. Timeouts, runtime errors and some external network results also need review rather than automatic attribution to the docs.
Triage installation failures
A useful failure report identifies the exact command, the environment and the first error, then connects those details to a change a reader can use. Avoid updating instructions based on a transient failure until the cause is understood. Equally, do not dismiss a reproducible missing package or mismatched import as a generic environment issue.
| Symptom | Likely cause | What to change in the docs |
|---|---|---|
| Package name cannot be found | The documented name is wrong or the distribution path changed | Verify the intended distribution and correct the name or source |
| Install succeeds but the first import fails | The page uses the wrong import, artifact or environment | Correct the first-use example or clarify which artifact and interpreter it requires |
| An optional feature fails after a basic install | The relevant extra or prerequisite is missing | Document the needed extra or prerequisite next to that feature |
| A requirements file is not found | The command assumes a different directory or a file not provided to readers | State the working directory and provide or reference the correct file |
| An editable install fails from the shown location | The page assumes a checkout or project directory that was not established | Explain how to obtain the checkout and where to run the command |
npm ci rejects project files |
The lockfile and package.json dependencies do not match |
Correct the project files and ensure the instructions point to the matching checkout |
| A command is unavailable | The image or reader environment lacks a required tool | State the prerequisite and use a test image appropriate to the stated path |
| Download or network access fails | Network access or the external package source is unavailable | Clarify network requirements if missing; investigate the service before changing a correct package name |
| The first call requires a credential | The example cannot complete with the test's available inputs | Explain the credential prerequisite and record the untested authenticated step |
Retest after changing the page or the artifact. Where a page offers different install methods, rerun the affected alternatives independently. If a failure depends on a particular runtime or operating system, preserve that environment in the reproduction so reviewers can see the scope of the fix.
Installation testing checklist
| Check | Evidence to retain |
|---|---|
| Identify the intended reader and starting environment | Runtime, platform or image, working directory and prerequisites |
| Copy the documented command exactly | Package name, extras, flags, file paths and package source |
| Start with a clean environment | How it was created and what the image already contained |
| Test each important alternative separately | Which method and environment each result covers |
| Include the documented requirements, constraints or lockfile | The files and project configuration used by the test |
| Run installation followed by first use | Command, import or invocation and its actual result |
| Compare promised output | Stable expected content and any explained variable values |
| Record test-only setup and network assumptions | Setup commands, image and whether network was available |
| Classify the failure before editing | Error, likely cause, decision and responsible source |
| Rerun after the fix | The same path and environment, with remaining skips visible |
Use the docs testing checklist to assign these checks to changes and releases. For a site built from Markdown, the MkDocs testing guide explains how tabs, included files and generated reference pages affect which install command readers see.
What this does not replace
A single installation check does not establish support across every operating system, runtime, package source or configuration. Choose a representative set of environments based on the promises in the docs, and state what each result covers. Review the prose for missing prerequisites, clarity and the order in which a new reader encounters the steps. Keep build, link and usability checks alongside execution.
DocsAlive executes supported shell blocks, but it has no dedicated support for operating-system package managers. A Homebrew or apt command in a shell block runs only if the selected image provides that tool, and commands that need host-level or root changes may fail inside the sandbox, which runs as an unprivileged user with a read-only root filesystem. Go, Rust, Java and other compiled-language examples are not executed by the product. Full TypeScript and JSX are not judged. Examples requiring credentials are classified rather than executed with real keys; customer-provided test credentials are not currently supported. Use separate appropriate checks for unsupported paths.
For a starting point, the free health check lets you send a public repository you own or maintain; the team runs the engine on up to 40 of its docs pages and emails the real report within 2 business days; the checkout is deleted after the run. Pricing is Open source $0/month, Team $299/month, Growth $699/month, and Scale from $1,499/month.
The sources we checked
| Source | What it covers |
|---|---|
| pip install reference | Requirement inputs, requirements files, editable installs and constraints |
| Python Packaging User Guide | Installing Python packages and optional dependencies |
Python venv documentation |
Creating virtual environments and default package isolation |
| npm install | Installing packages using package specifications |
| npm ci | Lockfile requirements and clean project installation behavior |
| Docker container run | Running a command in a new container and --rm cleanup |
FAQ
What should an installation documentation test include?
Start in the environment the page describes, run the exact install command, and follow it with the first documented import or command. Record the package source, runtime, files, setup and result so reviewers know what was tested.
Why test an import after pip install?
A successful install does not by itself establish that the page's next import works in the interpreter the reader will use. Run both steps in the same environment and investigate any mismatch in package name, import name, artifact or prerequisites.
Should I use npm install or npm ci in the test?
Use the command the reader is instructed to run. npm ci requires an existing lockfile, installs the project as a whole and fails if the lockfile and package.json dependencies disagree. A separate package-addition path needs its own test.
Does a virtual environment reproduce a clean machine?
It provides an isolated Python package environment by default, but it does not recreate an entire operating system or supply every system tool. Record those other prerequisites and test the relevant platform separately.
Can I provide setup before each page?
Yes. The product's --setup CMD runs a shell command at the start of each page's container and can be repeated. Record it, because setup can conceal an installation instruction that a reader would otherwise need.
What does a missing-install-step hint mean?
It indicates that a page imports a package the page never installs. The product reports it as a hint rather than automatically filing a docs bug. Review the page's stated prerequisites and decide whether an install instruction is missing.
Can the product test apt or Homebrew instructions?
Only as ordinary shell blocks. Such a command runs only if the selected image provides the tool and the sandbox permits the operation, and the sandbox runs as an unprivileged user with a read-only root filesystem. There is no dedicated operating-system package-manager testing; use a separate suitable environment for those paths.
Does a successful container test cover every platform?
No. It covers the path in the selected image with the inputs and network conditions used for that run. Test additional environments when the documentation promises support for them, and record any skipped or credential-dependent steps.
