Open benchmark loss ledger for the 180-task public run.

By category — what the pattern says

P2 reference finding — 17 losses, mostly to smart-grep

This is sverklo's largest loss surface and the most-fixable. Smart-grep wins on tasks where the symbol appears in a flat-namespace context (lodash, express) because literal string match with definition-pattern filtering is genuinely the right tool. Sverklo's hybrid (BM25 + embeddings + PageRank) over-thinks these. Mitigation paths under investigation: route flat-symbol P2 queries to a regex shortcut, or weight the lexical channel higher on single-token queries.

P1 definition lookup — 11 losses, split between fastapi (5) and lodash/sverklo/requests (6)

The fastapi P1 losses are correlated with the audit's Python-decorator gap (now fixed in v0.20.19, but P1 specifically is a separate retrieval path — these may persist). The other 6 are mostly multi-line class/function definitions where the chunker's line cap truncates the relevant span. The parser fix in v0.20.17 (paren-aware brace counting) recovered ~464 references on sverklo's own repo but doesn't address the chunk-truncation pattern these P1 tasks hit.

P5 dead-code detection — 10 losses (now closed on 2026-05-13)

Two patterns, both now resolved. The fastapi cluster (5 tasks, all scored 0.00 on May 12) was the audit's DECORATOR_ENTRY_POINT regex being TS/NestJS-only — every FastAPI route method fell through as a false-positive orphan. v0.20.19 added Python decorator coverage; the 2026-05-13 rerun confirms these tasks now score 1.00. The sverklo-self P5 losses (5 tasks where sverklo scored 0.00 vs jcodemunch's 0.04) are also closed — sverklo now ties the field on P5 at 0.83 overall. The interesting P5 story is now jcodemunch: after the sverklo-bench#3 baseline refresh removed the max_results=100 cap, jcodemunch's P5 recall jumped to 1.00 across all 30 tasks (precision 0.34, F1 0.34, tokens 10,172). It's now the only baseline with zero false negatives on dead-code.

P4 file dependencies — 4 losses

The smallest loss surface. Sverklo wins P4 overall by 44 points over the next baseline (0.84 vs smart-grep 0.40). The 4 losses are edge-case files with dynamic imports or test-only fixtures where the import graph is genuinely ambiguous. We don't have a near-term fix for these — they require runtime tracing, not static analysis.

Every loss, sorted by F1 delta

Source: benchmark/results/2026-05-12T17-11-45-853Z/raw.jsonl in github.com/sverklo/sverklo. The 138 tasks not listed here are tasks sverklo either won outright or tied. The bench harness writes raw, summary, and report files on every run — reproducible with npm run bench:quick from a clean clone.

Why publish this

Most public benchmarks for AI tooling show wins only. The bench's wins live on /bench/ and /mcp/. This page is the half that doesn't get published anywhere else: where we're worse, by how much, and what we know about it.

Three reasons it has to exist:

• It tells you what sverklo is bad at before you install. If your work is mostly grep-shaped, flat-namespace, single-symbol P2 reference queries, smart-grep wins a meaningful slice of those tasks. That's a real signal — pick the right tool.
• It separates fixed-by-shipping from fundamental. The 5 fastapi P5 tasks scored 0.00 because of a Python-decorator gap in the audit's orphan detection. v0.20.19 ships the fix. The next bench run will move those to 1.00. P2 smart-grep losses are different — they're architectural, not bugs.
• It is the only thing keeping the wins credible. "F1 0.58 leader on a 180-task bench" reads as marketing without this page. With this page, the wins are reviewable.

Reproduce

git clone https://github.com/sverklo/sverklo && cd sverklo
npm install && npm run build
npm run bench:quick  # ~30-45 min, 180 tasks × 5 baselines
# Outputs: benchmark/results/<timestamp>/{raw.jsonl, summary.json, report.md}

Raw artifact for this page: 2026-05-12T17-11-45-853Z. Bench methodology: METHODOLOGY.md.