Process piece
Auditing your own tool with adversarial agents.
Every benchmark a tool author runs on their own tool flatters it. You wrote the tasks, you know the happy paths, and you stop investigating once the numbers look good. So before Miller replaced its predecessor, independent agent reviewers were pointed at the code and asked, in effect, to prove the replacement should not happen. A tool the author believed was finished gave up twelve prioritized defects, two of them P0.
Nothing about the method is specific to Miller. You need a candidate tool, a comparison target, agent sessions with no history, and the discipline to accept nothing without evidence. This page records the process so someone else can run it.
The setup
A fresh reviewer for every tool, with read-only access.
The audit compared Miller tool by tool against the tool it aimed to replace: one broad review of
product boundaries and cross-cutting risks, then nine separate reviews, one per MCP tool, each
against its counterpart. Every review ran in an ephemeral session with no persistence, no memory
of earlier passes, and strict MCP isolation. Reviewers had Read, Grep,
and Glob and nothing that could run or index either product. The broad review did
not count toward any tool-specific pass; each tool got its own fresh reviewer.
Isolation matters in both directions. A reviewer with session history inherits the author's framing, and a reviewer with write or execute access starts "fixing" instead of auditing. The comparison target was never built, indexed, or modified during the audit either; its behavior was established from source and recorded artifacts, so the audit could not disturb the thing it was measuring against.
The rules
No claim survives without local evidence.
Reviewer output was treated as a set of hypotheses to check. No claim was accepted, flattering or damning, until local source, artifact, telemetry, or observed behavior supported it, and every material claim was classified as accepted, corrected, rejected, or unproven. The rejected and corrected buckets matter most here. Agent reviewers misread code with complete confidence, and an audit that transcribes reviewer output unverified just swaps one bias for another.
The bar for "Miller replaces the old tool" was written down before the audit started: the replacement stands only when an agent obtains the evidence needed to act correctly with equal or better relevance, fewer calls, fewer returned tokens, lower wall-clock time, explicit uncertainty, and no correctness loss. Having the feature is not parity. "We also have a rename tool" means nothing if the rename can touch a homonym the old tool would have excluded.
What it caught
Twelve real defects in a "finished" tool.
The worst finding was structural. Reference workflows resolved a symbol to an exact identity and then threw the identity away, querying references by bare name, so four tools could return homonym results and a rename could over-reach. It had survived every author-run benchmark because the benchmarks never asked a homonym question. Other accepted findings included hard failures returned as successful plain text (automation could not tell success from failure), context bundles that omitted implementation bodies, and payloads that bounded rows but not delivered tokens.
Every finding got a written disposition, the fixes became a phased remediation program, and each tool was re-audited in another fresh session before its finding closed. The full audit matrix, source proofs included, is committed in the repository.
Where it fits
What the audit can and cannot tell you.
The audit answers whether the tool is correct and complete enough to ship. It cannot answer whether the tool is worth installing. That takes measurement against the baseline users actually have, which for a code tool in 2026 is a capable agent with shell access rather than a rival product. After remediation, Miller ran that experiment too: same agent, same tasks, same budgets, with and without the tool. The agent got 2.2× more tasks right with Miller and halved the wrong-action rate, from 27% to 13%; at the doubled budget it made no wrong-action errors at all. The experiment was rerun on Miller v1.22.1 on 2026-08-25 with the same tasks, repositories, model, seed, and budgets, and the 2.2× result held. The details, including what the rerun did not reproduce, are on the benchmark page.
To run this on your own tool: write the replacement standard down first, counting calls, tokens, and correctness rather than features. Run one broad review plus one per surface, each in a fresh, read-only, isolated session. Treat reviewer claims as hypotheses and classify each one against local evidence. Turn the accepted findings into a remediation program and re-audit each surface before closing it. Then benchmark against the honest baseline. None of this needs special tooling, just the willingness to let a reviewer argue against shipping.
The artifacts
Everything is committed.
# the audit matrix, source proofs, and dispositions
docs/findings/2026-07-22-miller-julie-takeover-matrix.md
# the paired benchmark that followed it
docs/findings/2026-07-29-miller-vs-bare-agent-calibration.md
docs/findings/agent-efficiency/2026-07-29-bare-agent/
# the 2026-08-25 rerun on Miller v1.22.1
docs/findings/2026-08-25-miller-vs-bare-agent-v1.22.1-calibration.md
docs/findings/agent-efficiency/2026-08-25-bare-agent-v1.22.1/
- The audit's rejected-claims record is preserved alongside the accepted findings; the misfires are part of the evidence.
- The benchmark harness is in-repo (
scripts/bench-agent-efficiency.py) so the follow-up experiment can be rerun. It was rerun on 2026-08-25, and both measurements are kept. - The extraction layer under Miller has its own depth story: hand-written extractors, not query files.