Smart-grep is sverklo's closest competitor on raw token cost. Sverklo at 652 input tokens per task, smart-grep at 714 — a ~9% advantage. On every other axis the gap is wider, and on the F1 axis specifically it grew from 7 points at 90-task scale to 24 points at 180-task scale (0.58 vs 0.34). The honest story: when is a tuned grep good enough? When isn't it? Numbers from the public 180-task bench across express, lodash, sverklo, requests, flask, and fastapi.
| Metric | Sverklo | Smart-grep | Gap |
|---|---|---|---|
| Overall F1 | 0.58 | 0.34 | +24 pts (was +7 at 90-task) |
| P1 definition lookup | 0.63 | 0.20 | +43 pts |
| P2 reference finding | 0.27 | 0.20 | +7 pts |
| P4 file dependencies | 0.84 | 0.40 | +44 pts |
| P5 dead-code detection | 0.83 | 0.83 | tie (v0.20.19 fix landed) |
| Input tokens / task | 652 | 714 | −62 tokens (~9% sverklo win) |
| Tool calls / task | 1.0 | 3.2 | 3.2× fewer |
| Wall time / task (warm) | 64 ms | 1,130 ms | 18× faster |
| Cold-start cost | ~26 s (index build, M-series) | 0 | Smart-grep wins on cold-start |
| Sverklo | Smart-grep | |
|---|---|---|
| Implementation | MCP server + tree-sitter + ONNX embeddings + PageRank + SQLite | Shell baseline: rg with language filters, definition-shaped patterns, ±10-line context reads |
| Setup | npm i -g sverklo && sverklo init (one-time index build) |
Install ripgrep; nothing else |
| Cold-start cost | ~180 s on a ~4000-file repo (index build) | None |
| Warm wall time | ~68 ms per task | ~1,130 ms per task (multi-grep cascade) |
| Cross-file graph awareness | Yes — symbol graph, PageRank-ranked retrieval, file deps | No — each grep call is stateless |
| Memory across sessions | Bi-temporal SHA-pinned memory | None |
| Languages | 10 first-class (TS/JS, Python, Go, Rust, C#, Vue, Markdown, Jupyter, etc.) | Any language grep matches (all of them); no symbol-level semantics |
| License | MIT | BSD-3 (ripgrep); the smart-grep wrapper used here is shipped in benchmark/src/baselines/smart-grep.ts MIT |
/compact — past decisions, file-level notes, pinned context
The "smart-grep" baseline in sverklo-bench is not naïve ripgrep. The implementation in benchmark/src/baselines/smart-grep.ts applies language-aware filters (file globs by extension), constructs def-shaped patterns per task type (e.g., function\s+name, class\s+name, def\s+name), reads ±10 lines of context around each hit, and de-duplicates. It is, in practice, the best version of "grep + a thoughtful prompt" — which is why we use it as the strong text-search baseline instead of a strawman.
Sverklo uses a hybrid pipeline: SQLite + sqlite-vec for storage, tree-sitter + regex fallback for parsing, BM25 + ONNX (all-MiniLM-L6-v2) embeddings for ranking, and PageRank over the import/symbol graph for centrality. The graph is what lets P4 ("which files depend on X?") return 0.84 instead of the 0.40 a grep wrapper can manage. Everything runs in-process from a single npm install; no daemons, no separate LSP processes.
Where sverklo lost in earlier runs: P5 (cross-file dead-code) used to show smart-grep at 0.83 vs sverklo at 0.67 because the audit's DECORATOR_ENTRY_POINT regex was TS/NestJS-only and flagged every FastAPI/Flask route as dead. Fixed in v0.20.19; the 2026-05-13 rerun shows both baselines tied at 0.83. We left the historical context here so the bench-loop arc is visible — the public losses page lists every task where sverklo still loses to at least one baseline.
MIT licensed. One command. Works with every MCP client.
click to copy