Why we gave our docs assistant a filesystem

A5kAI is the assistant that answers questions about AGNT5’s own docs. We could have embedded a third-party documentation bot and moved on. Instead, we built it ourselves because a docs assistant is part of the product surface. It needs to read the same docs we ship, cite the same URLs users can open, stream through our own API, and follow retrieval rules we can inspect and change.

The core idea is simple: keep semantic search for questions where meaning matters, but give the agent a second way to inspect the docs like an engineer would. Engineers grep for identifiers, open matching pages, scan nearby files, and verify against the current source. A5kAI does the same thing over the docs bundle that ships with the deployment.

Why build it ourselves

Documentation assistants look generic until they are wrong in ways users can see. We wanted control over five things: how the docs are synced, how retrieval is routed, how citations are selected, how the browser widget streams responses, and how regressions are evaluated before changes ship.

Owning that stack let us make a product-specific tradeoff. AGNT5’s docs corpus is small: a few hundred Markdown/MDX files, organized into known collections. That is small enough to download onto the worker and search directly. We did not need a container per chat session, a virtual filesystem backed by the vector database, or a separate hosted platform interpreting our docs for us. A real local folder was enough.

The architecture

A5kAI has two retrieval surfaces over the same documentation. The ingestion path chunks the docs, embeds them, and syncs them into a Pinecone hybrid index for semantic retrieval. The chat path downloads the CI/CD-published S3 docs bundle onto the worker, under a single docs root.

Two retrieval surfaces feed the same citation guard. The sync path downloads the published docs bundle onto the worker; the index path embeds the same docs into Pinecone. At query time the agent picks a tool per question, and the dashed lines show each tool reading back from the store its path built.

At answer time, the agent can use either surface. Semantic retrieval finds conceptually relevant sections. Filesystem retrieval runs bounded, read-only commands over the local docs folder. Both tools return the same envelope — {content, sources, error} — so the rest of the system does not care whether evidence came from Pinecone or from a shell read.

This is the part we like about building on AGNT5: the assistant is not a one-off web endpoint with hidden state. It is a normal hosted agent, with explicit tools, a deterministic post-processing callback, and the same deployment path as the rest of our agent workloads.

Why a real filesystem worked

The small corpus is the reason this design is solid. Recursive grep over a few hundred text files is fast. Reading a whole page is cheap. Returning the first few dozen lines of related pages is predictable. The command output is small enough to cap aggressively without losing the usefulness of the tool.

That changes the engineering calculus. At very large scale, faking a filesystem over an existing search database can be the right move. Mintlify described that approach for their assistant: they translate shell-like operations into Chroma queries to avoid spinning up real sandboxes for every session. Our constraints were different. For our docs, real files on the worker were simpler, faster to reason about, and easier to secure.

The shell boundary

The agent does not get a general shell. It gets a narrow docs-search tool implemented with real Unix commands: grep, cat, find, head, tail, wc, and a few other read-only utilities. Every command is validated before execution.

• Only known read-only binaries are allowed.
• Every path is resolved inside the docs root.
• Redirection, mutation flags, shell variables, loops, and execution escapes are blocked.
• Each command has a short timeout and capped output.
• Host paths are stripped before output reaches the model.

This keeps the tool useful without turning it into infrastructure risk. The model can inspect documentation, but it cannot mutate files, read the host filesystem, or run arbitrary programs.

# 1. binary must be in the read-only allowlist (grep, cat, find, head, ...)
# 2. every path argument must resolve inside the docs jail
# 3. no writes: no redirection, no exec/delete-style escape hatches
# 4. scrubbed environment + wall-clock budget + output cap
from agnt5 import tool

@tool
def execute_docs_command(command: str, docs_root: Path) -> dict:
    argv = validate_read_only_binary(shlex.split(command))
    reject_forbidden_flags(argv)
    reject_redirection_and_chaining(argv)

    for path in paths_in(argv):
        resolved = (docs_root / path.lstrip("/")).resolve()
        if not resolved.is_relative_to(docs_root):
            return {"error": "path outside docs corpus"}

    output = run_with_timeout(
        argv, cwd=docs_root, env={"LC_ALL": "C"}, seconds=COMMAND_BUDGET_SECONDS
    )
    return {
        "content": scrub_host_paths(output)[:MAX_OUTPUT_CHARS],
        "sources": pages_mentioned_by(command, output),
        "error": None,
    }

Routing

Shell-first does not mean shell-only, and it does not mean vector search is wrong. It means the agent starts with the cheapest source-level read that could ground the answer, then uses semantic retrieval when the question needs conceptual matching.

This is the same workflow engineers use manually. If the user asks about an exact symbol, command, environment variable, page title, or known product term, search the files directly. If the user asks a fuzzy “how do I…” question, the shell pass can be small and semantic search does the heavier lifting. The point is not to replace RAG; it is to keep RAG from being the only way the assistant can see the docs.

Filesystem search gives the agent a current, exhaustive view of the docs tree. Semantic search gives it recall over phrasing and concepts. The assistant needs both.

We also enforce the route in code. If the model calls semantic search before it has made a filesystem read in the current turn, the tool does not execute. It returns an internal steering note telling the agent to use the shell first.

Citations

Owning the assistant also let us make citations mechanical. Chunks are cut at real heading boundaries, and file paths map back to the same URLs users see on the docs site. When the model touches a page through the shell, that page becomes a candidate source. When semantic retrieval returns a chunk, its metadata becomes a candidate source.

The final answer does not get to invent its own bibliography. The system selects sources from pages actually retrieved or read, then ranks them against the answer’s distinctive vocabulary: identifiers, flags, commands, backticked terms, and other rare tokens. The result is a citation list tied to evidence, not a list of links the model thought looked plausible.

What we measured

We measured A5kAI with two AGNT5 eval datasets, both built from the same 29-question suite.

• A comparison eval checks the user-visible result: answer correctness, citation validity, and whether the expected source page was cited.
• A trajectory eval checks the agent’s behavior: which tools it called and whether it stayed within the tool-call budget.

The important split is outcome versus trajectory. A docs assistant can occasionally land on the right sentence through a brittle path. We wanted the evals to catch that, because retrieval quality is not just what the model says at the end; it is also whether the system used a route that will keep working when the docs change.

The suite covers ten classes of questions we saw break in practice: aggregation, membership, entity counts, negative existence, exact identifiers, conceptual how-tos, named-page reads, disambiguation, follow-ups, and scope control. That gave us cases like:

• Aggregation: “how many templates does AGNT5 have?” Correct answer: exactly 7, with no invented templates.
• Membership: “which templates have Slack integration already?” Correct answer: zero templates, even though Slack appears elsewhere in the docs.
• Entity count: “how many AI providers does AGNT5 support today?” Correct answer: count the providers listed in the integrations page, not retrieved snippets.
• Negative existence: “how do I install the AGNT5 Ruby SDK?” Correct answer: the docs do not mention a Ruby SDK; do not invent install steps.
• Exact identifier: “what is RetryPolicy?” Correct answer: cite the real retry configuration object and its documented fields.
• Follow-up: “can you explain each?” after a template-count answer. Correct behavior: resolve “each” from history and re-read the known template pages.

We used four scorers. answer_correctness is an LLM judge that compares the answer to a reference answer with a negation-safe rubric. citation_validity checks that every cited AGNT5 URL resolves to a real docs page. gold_page_cited checks that at least one expected evidence page appears in the citations. trajectory_discipline checks whether the required tools were called within the budget.

The trajectory scorer changed how we improved the system. It let us fail a run where the answer looked plausible but the agent used semantic search for a collection count, skipped the filesystem proof for an absent feature, or spent too many tool calls rediscovering pages already named in the conversation.

Against the 29-question gate, the filesystem-backed version passed 26 items, up from 20 for the hybrid-only version.

Treat the numbers as project-specific, not universal benchmarks. The reusable result is the evaluation shape: answer quality, citation validity, expected-source coverage, and tool trajectory all measured separately. That separation made regressions obvious. A model could cite a valid page and still miss the gold page. It could answer the final text correctly while using a brittle route. The evals told us which problem we were fixing.

Where this leaves us

Building our own assistant was not about rejecting third-party docs platforms on principle. It was about owning the parts that determine answer quality: the source snapshot, the retrieval policy, the citation machinery, and the evaluation loop.

For a small docs corpus, a worker-local filesystem is a powerful primitive. Let CI/CD publish a docs bundle to S3, have the deployment download that bundle onto the worker, jail a small read-only command set to that folder, and return shell results through the same evidence pipeline as normal RAG. That is the pattern that made A5kAI work.

A5kAI is open source and runs on AGNT5. It combines a hosted docs agent, a hybrid retrieval index, and a worker-local filesystem view downloaded from the CI/CD-published S3 docs bundle.