Real runs on three open-source docs sites.
Before asking anyone to trust DocsAlive, we ran it on three popular open-source documentation sites and published the numbers, the bugs and where the engine got it wrong. Project names are withheld until we have reported the findings to their maintainers.
Method
Unmodified repos
Public default branches as of October 19, 2026. No docs were edited.
Current releases
Each page installs the latest published package, like a reader following the docs today.
Deterministic
Triage only. No AI model involved in any number on this page.
Hand-checked
Bugs marked "verified" were re-checked by a person against the source and the sandbox output.
Project A
a popular open-source Python HTTP clientmkdocsPythonEngine v0.2.1 · run October 19, 2026 · 38s · setup: pip install -q <package>
Categories
| Engine-flagged docs bug | Status |
|---|---|
| extensions.md:156 api-drift: parameter `extensions` is not accepted by the current API |
not yet reviewed |
| extensions.md:174 api-drift: parameter `extensions` is not accepted by the current API |
not yet reviewed |
| transports.md:180 invalid-syntax: is not valid code: invalid syntax. Perhaps you forgot a comma? |
not yet reviewed |
| quickstart.md:118 stale-output: ran, but the documented output is Python 2-era (u'' strings) and no longer matches |
not yet reviewed |
Project B
an open-source AI SDK with a large docs sitemintlifyPython / JSEngine v0.2.1 · run October 19, 2026 · 290s · setup: (per-page installs)
Categories
OpenAPI docs/openapi.json: 68 operations, 87 examples, 0 broken $refs.
| Engine-flagged docs bug | Status |
|---|---|
| vllm.mdx:135 invalid-syntax: is not valid code: illegal target for annotation |
not yet reviewed |
| gemini-3-with-<project>-mcp.mdx:73 api-drift: `MCPServerHTTP` no longer exists in `pydantic_ai.mcp` |
not yet reviewed |
| llamaindex-react.mdx:78 api-drift: `FunctionCallingAgent` no longer exists in `llama_index.core.agent` |
not yet reviewed |
| llama-index.mdx:38 api-drift: parameter `org_id` is not accepted by the current API |
not yet reviewed |
| llama-index.mdx:164 api-drift: `FunctionCallingAgent` no longer exists in `llama_index.core.agent` |
not yet reviewed |
| oss-v2-to-v3.mdx:210 api-drift: parameter `org_id` is not accepted by the current API |
not yet reviewed |
| migration_guide_template.mdx:104 invalid-syntax: is not valid code: invalid syntax. Perhaps you forgot a comma? |
not yet reviewed |
Project C
a widely used open-source JavaScript developer tooldocusaurusJavaScriptEngine v0.2.1 · run October 19, 2026 · 11s · setup: npm install <package>
Categories
What these numbers don't say
- A high fail count isn't a quality judgment. Most docs examples intentionally depend on earlier context or real credentials.
- "Likely docs bug" is the engine's classification. We list which ones a person has verified and which haven't been reviewed yet.
- These projects didn't ask to be tested. We plan to report each verified finding to the maintainers before naming any project. Nothing here is a criticism of their work.