Benchmarks
Measured, not estimated.
Every number here comes from A/B runs: the same agent on the same task, once with Scubiee and once with native search tools only.
Average saving
30%+
fewer agent tokens on average
Median 32%, mean 39% per task, across all 11 tasks we measured on Claude Code and Kiro, including one where Scubiee cost more.
- 61%
- fewer tokens on multi-file dev tasks
- Claude Code · 9 tasks, 67 runs
- 66%
- fewer tokens on hard cross-module changes
- Claude Code · 5 tasks, 34 runs
- 4x
- cheaper on a cross-module integration task
- Kiro CLI · 3 runs per arm
- 5x
- fewer greps, ~2x fewer tool calls on hard tasks
- Claude Code · average per task
Claude Code
Tokens per task run, without vs with Scubiee
Agent: Claude Code (Claude Sonnet) via Claude Agent SDK. Unit: tokens per task run (input + output + cache reads + cache writes). Token counts are the SDK's real usage, not estimates. Both arms use the same native tools (Read, Grep, Glob, Edit); the with arm also has the Scubiee MCP server.
| Task group | Tasks | Runs (with / without) | Without | With | Saved |
|---|---|---|---|---|---|
| All multi-file dev tasks | 9 | 31 / 36 | 730,980 | 287,629 | 61%2.54x |
| Hard cross-module changes | 5 | 17 / 17 | 939,252 | 316,681 | 66%2.97x |
| Simpler single-file changes | 4 | 14 / 19 | 544,631 | 252,351 | 54%2.16x |
Average tokens per task run.
Every task, including the ones where Scubiee helped little
| # | task | category | files involved | without | with | delta | ratio | saved |
|---|---|---|---|---|---|---|---|---|
| 01 | New sync-strategy tier | hard | freshness + sync status | 2,596,365 | 435,092 | −2,161,273 | ██░░░░░░░░░░0.17 | −83% |
| 02 | Add importers alias to expand | hard | locate + trace | 426,064 | 163,621 | −262,443 | █████░░░░░░░0.38 | −62% |
| 03 | Unify CPU-thread budget | hard | memory budget + embedder | 775,718 | 404,388 | −371,330 | ██████░░░░░░0.52 | −48% |
| 04 | Net chunk delta in sync result | hard | incremental + sync loop | 355,826 | 243,153 | −112,673 | ████████░░░░0.68 | −32% |
| 05 | Consistent safety-pause check | hard | incremental + CLI | 119,612 | 95,810 | −23,802 | ██████████░░0.80 | −20% |
| 06 | Surface dirty count in timings | hard | freshness + engine | 305,615 | 269,546 | −36,069 | ███████████░0.88 | −12% |
| 07 | Root-hash fingerprint | two-file | — | 275,726 | 222,343 | −53,383 | ██████████░░0.81 | −19% |
| 08 | Total-changed field | two-file | — | 211,608 | 226,242 | +14,634 | ████████████1.07 | +7% |
Values are average tokens per run for each arm. ratio is with ÷ without; below 1.00 means Scubiee used fewer tokens.
hard: needs edits in two or more files in different subsystems, where the second file can only be found by following the call graph, not by searching for one word. two-file: an edit on a well-defined seam between two files.
Add importers alias to expand was built after the first five hard tasks, in a subsystem not tested before, to check the savings reproduce on a fresh task. They did: 2.6x cheaper.
Total-changed field targets exact names that plain text search finds in one hit, so Scubiee had nothing to add and cost 7% more. It stays in the average.
Why it saves tokens
The agent stops grep-hunting
Tool calls
18.4→9.5
per hard task, without → with
Grep searches
7.2→1.3
per hard task, without → with
File reads
6.6→2.6
per hard task, without → with
Worst single run
2.15M→~578k
tokens, same task
Without Scubiee, agents search, read the wrong file and search again. Scubiee hands back the relevant code and its callers in one or two calls. The harder the code is to find, the bigger the saving: exact-name edits save 10–20%, hard cross-module changes 50–80%.
Kiro CLI
Credits per task run, without vs with Scubiee
Agent: Kiro CLI v2.23.1 (headless, model auto), run as kiro-cli chat. 3 tasks, 22 runs. The without arm uses Kiro's native tools; the with arm adds the Scubiee MCP server (map with find / focus) and its rules. Kiro reports cost in credits, its billing unit, so these numbers are not comparable with the Claude Code token counts above.
| Task | Type | Runs per arm | Without | With | Saved |
|---|---|---|---|---|---|
| Sync severity roll-up | integration | 3 | 4.31 | 1.08 | 75%4x |
| New health CLI command | long two-turn feature build | 4 | 11.24 | 4.01 | 64%2.8x |
| Unify token estimate | cross-module edit | 4 | 3.70 | 2.85 | 23% |
| Average | 6.42 | 2.65 | 54%per task |
Average credits per task run.
What changed in the agent's behaviour
| task | without | with |
|---|---|---|
| Sync severity roll-up | 7 | 2 |
| New health CLI command | ~16 | ~10 |
| Unify token estimate | 4.25 | 2.5 |
- The Scubiee runs on the integration task were tight (0.96–1.19 credits); the native runs were all above 4 credits.
- Unify token estimate saved 23% on average and 40% at the median; one costly run pulls the mean down.
- On the long feature build, Scubiee also finished about 35% faster (about 205 s vs 310 s wall time).
- Kiro used Scubiee in every run: the map tool replaced native searching rather than adding to it.
How we measured
Methodology and limits
Methodology
- Same prompt, model, permissions and byte-identical starting code in both arms; only Scubiee differs
- Every run graded by hidden deterministic tests the agent never sees
- Claude Code: shell disabled in both arms; $4 cap per run (understates the no-Scubiee cost)
- Real changes to a 130+ module Python codebase
- Hidden tests were checked to fail on the original code and pass on a correct change.
Limits
- Internal benchmark on one codebase, not third-party
- 2-5 runs per task per arm; individual percentages are not tight estimates
- Small or zero saving on trivial exact-name edits
- Kiro credits and Claude Code tokens are different units
Check the raw numbers
The per-task tables and the machine-readable data behind this page are in the repository.
Internal benchmark: same agent, prompt and code with and without Scubiee, graded by hidden tests. Results vary by task.