bench:primitives — 90-task code retrieval evaluation

Five retrieval baselines, three real OSS codebases (express, lodash, sverklo), 90 hand-verified tasks. Reproducible. The naive baseline is what most agents do today — blind grep, 20K tokens of unranked regex hits per task.

May 4, 2026 PM — post-fix table (sverklo v0.20.2)

Note: sverklo numbers are from the post-v0.20.2 single-baseline rerun (2026-05-04T19-38-11-592Z); other baselines are from the morning 5-baseline run (2026-05-04T14-13-23-716Z) and didn't change between runs (no commits affecting them). Reproducible from a fresh clone with npm run bench:quick; full numbers should match within run-to-run variance.

What changed:

• Sverklo wins overall F1 (0.56), edging smart-grep (0.49). Pre-fix it was 0.45 (smart-grep led).
• P1 is now a tie at 0.73 between sverklo and jcodemunch. Pre-fix sverklo was at 0.30 — both projects had to ship parser fixes (sverklo for IIFE brace-counting, jcodemunch for the 500KB file cap) to land here.
• Sverklo still wins P4 (0.71), slightly down from 0.76 pre-fix. Tracked in #28.
• Token economy: sverklo at 469 avg input tokens, ~3× lower than smart-grep (1,220) and jcodemunch (1,267). Single tool call per task vs grep's 4–7.
• Smart-grep still wins P2 (0.40): tuned ripgrep on call-site references is genuinely competitive — sverklo at 0.25 here regressed slightly (#28).

How to reproduce: npm install -g sverklo@0.20.2 && cd /path/to/repo && npm run bench:quick. Bench harness lives at github.com/sverklo/sverklo/tree/main/benchmark. Issues #26, #27, #28 document the methodology iterations that produced this run.

May 4, 2026 AM — pre-fix table (sverklo v0.20.1, kept for diff)

Pre-v0.20.2 numbers preserved here so the diff between the two runs is auditable. Raw data at benchmark/results/2026-05-04T14-13-23-716Z/.

May 2026 — All 5 baselines

How to read this: No single baseline dominates. Different tools win different categories. The story isn't "sverklo beats everything" — it's "different retrieval substrates have different strengths, and the load-bearing axis depends on what you're optimizing for." Token economy + P4 are sverklo's clearest wins; P1 goes to jcodemunch; P2 to smart-grep.

Honest false positives we filed back to upstream:

• jcodemunch flags Express's createApplication as dead code — CommonJS module.exports = X isn't modeled as a use site, so the only export of the entire module appears to have no callers.
• gitnexus's impact analysis returns 0 affected modules for createApplication — same blind spot.
• Both tools' P2 implementations are import-graph-only — should be advertised more prominently in their docs.

Reproducer (requires uvx for jcodemunch and npm i -g gitnexus on PATH):

git clone https://github.com/sverklo/sverklo && cd sverklo
npm install && npm run build

# All 5 baselines
npm run bench:quick

# Single baseline
BASELINES=jcodemunch npm run bench:quick
BASELINES=gitnexus   npm run bench:quick
BASELINES=sverklo    npm run bench:quick

What this benchmark does NOT measure

• Real coding-task latency. 90 retrieval primitives, not 90 end-to-end agent runs. The token economy translates to real-world savings, but a follow-up bench:swe (65 SWE-bench-style questions × 5 OSS repos) measures that and is on the public roadmap.
• Anything beyond Express + lodash + sverklo. The current slice is three JS/TS codebases. Coverage on Go, Python, Rust is the next dataset extension.
• Cross-repo retrieval. The bench is single-repo; sverklo's workspace and cross-repo features aren't exercised.
• Cost of indexing. The 3690ms cold-start figure is the index build for sverklo. We list it as a separate column rather than amortizing it into wall time, so you can decide whether your usage pattern justifies it.

Methodology

• Tasks: 90 hand-verified retrieval primitives across expressjs/express, lodash/lodash, and sverklo/sverklo. Each dataset contributes 30 tasks distributed across P1 (definition lookup, 10), P2 (reference finding, 10), P4 (file dependencies, 5), P5 (dead-code detection, 5).
• Metrics: F1, recall, precision, total input tokens, tool-call count, wall time. tokens_per_correct_answer = input_tokens / max(recall, 0.01) — lower is better. The gated column averages only over runs where F1 ≥ 0.8 — refusing to reward "found nothing cheaply".
• Tolerances: P1 uses ±3-line tolerance, P2 uses ±2 lines, P4/P5 use set membership.
• naive-grep floor: grep -rn <sym> . then read top 10 files in full.
• smart-grep: language filters, ±10-line context reads, definition-shaped patterns. The strong baseline.
• sverklo: spawns the MCP stdio server once per dataset; cold-start is the index build.
• Run environment: Apple Silicon laptop, Node 25, sverklo running with --expose-gc, ONNX all-MiniLM-L6-v2 (~90 MB on disk).

Reproducing this

git clone https://github.com/sverklo/sverklo && cd sverklo
npm install
npm run build
npm run bench:quick                           # all baselines, all datasets
BASELINES=sverklo,jcodemunch npm run bench:quick   # single baseline filter
DATASETS=express npm run bench:quick               # single dataset filter

Raw outputs (raw.jsonl, summary.json, report.md) land in benchmark/results/<timestamp>/. The report.md mirrors this page's tables. Disagreements with our numbers are useful — file an issue with your machine spec and the run timestamp.

Submitting a new baseline

If you maintain a code-search tool, code-intelligence MCP server, or retrieval system, you can have it benchmarked here on the same task suite. Open a PR to sverklo/sverklo adding benchmark/src/baselines/<your-tool>.ts — auto-bench CI runs on the PR within ~10 minutes against the express dataset (~30 tasks) and posts a results-table comment back. You don't need to run the harness locally first; CI does it. Methodology repo: github.com/sverklo/sverklo-bench. Workflow source: .github/workflows/auto-bench.yml. Tracking: sverklo-bench#4.

Performance benchmarks (separate)

This page is the retrieval benchmark. We also publish performance numbers (cold index time, search latency, impact-analysis time) on five real OSS codebases at /benchmarks/. Both are reproducible from BENCHMARKS.md.

Cite this

If you reference this benchmark in academic work or comparison material:

@misc{sverklo_bench_primitives_2026,
  title  = {Sverklo bench:primitives — a 90-task retrieval evaluation for AI coding agents},
  author = {Groshin, Nikita},
  year   = {2026},
  doi    = {10.5281/zenodo.19802051},
  url    = {https://sverklo.com/bench/}
}

A few details we sweated

Hand-verified ground truth. Every one of the 90 task answers was inspected by hand at the fixed commit. Auto-generated ground truth from existing tooling correlates with whatever generated it; we wanted the harness to be cleanly testable against any retrieval system, including future ones we haven't built.

The losing slice gets the same prominence as the winning slice. The dead-code (P5) F1 = 0.02 number lives in the same table as the wins, two scrolls apart. A bench that only releases when the maintainer wins is marketing; a bench that releases when the maintainer loses is a bench. The contribution is the harness, not the leaderboard.

Tokens-per-correct-answer is the primary axis, F1 is secondary. Most retrieval evaluations report F1 first because they were designed for human-facing search. AI agents inside bounded context windows pay for every token returned; that opportunity cost compounds across an editing session. We report both axes and explain the tradeoff in plain terms; the agent-relevant axis is the one that earned the headline callouts above.

Naive grep is the floor, not the strawman. The naive baseline runs grep -rn <sym> . then reads the top 10 matching files in full — the same thing a Claude Code agent does on its first 5 minutes against a fresh codebase. If your bench's naive baseline scores 5% F1, you're probably measuring against a strawman; ours scores 0.35, which matches what real agents actually achieve on real tasks.

Cold-start is a separate column, not amortized. Sverklo's index build is 3,690 ms on this corpus. We list it as its own column rather than averaging it into wall time so you can decide whether your usage pattern justifies the upfront cost. For a 10-task session it dominates; for a multi-hour session it disappears.

Raw JSONL output, not just aggregates. benchmark/results/<timestamp>/raw.jsonl has every task's input, the system's output, and the per-task scoring breakdown. Disagreements with our aggregates are useful — file an issue with your machine spec and run timestamp and we'll triage.

Three codebases is still small. We say so. The "What this benchmark does NOT measure" section above isn't decorative. The next dataset extension is Go / Python / Rust on 5+ codebases. The current 90-task slice is what we have today, with the limitations explicitly listed.

Get started

If the token-economy numbers look interesting:

npm install -g sverklo
cd your-project
sverklo init

sverklo init auto-detects which AI coding agents you have installed (Claude Code, Cursor, Windsurf, Zed, Antigravity) and writes the right MCP config files. Back to the homepage →