Sverklo loses 42 of 180 bench tasks. Here are all of them.

By category — what the pattern says

P2 reference finding — 17 losses, mostly to smart-grep

This is sverklo's largest loss surface and the most-fixable. Smart-grep wins on tasks where the symbol appears in a flat-namespace context (lodash, express) because literal string match with definition-pattern filtering is genuinely the right tool. Sverklo's hybrid (BM25 + embeddings + PageRank) over-thinks these. Mitigation paths under investigation: route flat-symbol P2 queries to a regex shortcut, or weight the lexical channel higher on single-token queries.

P1 definition lookup — 11 losses, split between fastapi (5) and lodash/sverklo/requests (6)

The fastapi P1 losses are correlated with the audit's Python-decorator gap (now fixed in v0.20.19, but P1 specifically is a separate retrieval path — these may persist). The other 6 are mostly multi-line class/function definitions where the chunker's line cap truncates the relevant span. The parser fix in v0.20.17 (paren-aware brace counting) recovered ~464 references on sverklo's own repo but doesn't address the chunk-truncation pattern these P1 tasks hit.

P5 dead-code detection — 10 losses

Two patterns. The fastapi cluster (5 tasks, all scored 0.00) was the audit's DECORATOR_ENTRY_POINT regex being TS/NestJS-only — every FastAPI route method fell through as a false-positive orphan. Fixed in v0.20.19 with Python decorator coverage; the next bench rerun should move these from 0.00 to 1.00. The sverklo-self P5 losses (5 tasks, 0.00 vs jcodemunch's 0.04) are different: jcodemunch's barely-above-zero F1 wins only because sverklo's orphan list contains 10 items vs jcodemunch's empty. Both are wrong; the bench's empty-expected scoring favors the wronger one.

P4 file dependencies — 4 losses

The smallest loss surface. Sverklo wins P4 overall by 44 points over the next baseline (0.84 vs smart-grep 0.40). The 4 losses are edge-case files with dynamic imports or test-only fixtures where the import graph is genuinely ambiguous. We don't have a near-term fix for these — they require runtime tracing, not static analysis.

Every loss, sorted by F1 delta

Source: benchmark/results/2026-05-12T17-11-45-853Z/raw.jsonl in github.com/sverklo/sverklo. The 138 tasks not listed here are tasks sverklo either won outright or tied. The bench harness writes raw, summary, and report files on every run — reproducible with npm run bench:quick from a clean clone.

Why publish this

Most public benchmarks for AI tooling show wins only. The bench's wins live on /bench/ and /mcp/. This page is the half that doesn't get published anywhere else: where we're worse, by how much, and what we know about it.

Three reasons it has to exist:

• It tells you what sverklo is bad at before you install. If your work is mostly grep-shaped, flat-namespace, single-symbol P2 reference queries, smart-grep wins 17 of those 60 tasks. That's a real signal — pick the right tool.
• It separates fixed-by-shipping from fundamental. The 5 fastapi P5 tasks scored 0.00 because of a Python-decorator gap in the audit's orphan detection. v0.20.19 ships the fix. The next bench run will move those to 1.00. P2 smart-grep losses are different — they're architectural, not bugs.
• It is the only thing keeping the wins credible. "F1 0.56 leader on a 180-task bench" reads as marketing without this page. With this page, the wins are reviewable.

Reproduce

git clone https://github.com/sverklo/sverklo && cd sverklo
npm install && npm run build
npm run bench:quick  # ~30-45 min, 180 tasks × 5 baselines
# Outputs: benchmark/results/<timestamp>/{raw.jsonl, summary.json, report.md}

Raw artifact for this page: 2026-05-12T17-11-45-853Z. Bench methodology: METHODOLOGY.md.