measured 2026-10-10 / 11
What we measured, and what we did not show
Two statements are backed by the runs below. Each carries its conditions. Multi-hop answers were graded by a strict Sonnet judge (per the run notes; the model id is not logged in the result files): the same entity or value counts, hedging does not. LoCoMo answers were graded with the separate, more lenient grading prompt that is published for LoCoMo comparisons.
The two statements
- On 260 public multi-hop questions (wiki and news), with a small answer model searching once, kal search answered 7 points more often correctly than the same search with the graph turned off (66.9% → 73.8%, p = .001).
- On 100 MuSiQue questions, a Sonnet agent using four kal tools (via CLI) matched or slightly beat grep (80% vs 74%, p = .07), using 3.4 calls per question against grep’s 6.3; grep went over the ten-call instruction on 17 of the 100 questions. With a Haiku agent we found no significant difference (72% vs 75%, p = .55, in grep’s direction).
Search once, then answer
Answer model: Haiku unless noted (per the run notes; the model id is not logged in the result files). One search per question.
| Data | Plain RAG (kal, graph off) | Shipped kal_search | Passages provided per question |
|---|---|---|---|
| Multi-hop, 260 questions (HotpotQA · MuSiQue · MultiHop-RAG) | 66.9% | 73.8% (vs graph off: p = .001) | 83.1% |
| LoCoMo conv-26, 152 questions | — | 84.2% (37 time questions: 78.4%) | n/a |
| LoCoMo conv-26, Sonnet answering | — | 97.4% | n/a |
- Plain RAG is not another product: it is kal’s own BM25 + vector search with only the graph switched off.
- The last column is every passage the dataset gives for a question — the right ones and the distractors — not the right ones alone.
- LoCoMo does not mark passages per question, so that column is n/a. Given the whole conversation instead, Haiku answered 100% and Sonnet 99.3%.
- With a strong answer model and the full context, the differences disappear. The gain showed up with a small answer model and a single search.
Agents: grep tools vs kal tools
MuSiQue, 100 questions. kal side: four of the seven MCP tools (search, entity, neighbors, doc) through a CLI shim. grep side: grep, file listing and read. “Ten calls per question” was an instruction, not enforced — Sonnet + grep went over on 17 questions (max 22), Haiku + grep on 1 (max 11).
| Correct | 2-hop / 3-hop / 4-hop (n = 47 / 33 / 20) | Calls per question | Characters returned per question | |
|---|---|---|---|---|
| Haiku + grep | 75% | 83 / 70 / 65 | 3.7 | 10.8k |
| Haiku + kal | 72% | 79 / 73 / 55 | 3.5 | 26.7k |
| Sonnet + grep | 74% | 79 / 70 / 70 | 6.3 | 18.9k |
| Sonnet + kal | 80% | 87 / 73 / 75 | 3.4 | 33.3k |
| (reference) one kal_search | 58% | 66 / 64 / 30 | 1 | — |
| (reference) passages provided, at once | 78% | 83 / 79 / 65 | — | — |
- McNemar: Haiku kal vs grep 4:7 (p = .55) · Sonnet kal vs grep 7:1 (p = .07). Any agent beat one-shot search by a wide margin (e.g. Haiku + grep vs kal_search 23:6, p = .002).
- kal returns 1.8–2.5× more characters than grep. Characters are not tokens, and this does not show that kal saves context.
Limits
These corpora are short wiki passages full of proper nouns, which grep can find by exact name. Personal notes and Korean may behave differently. On the author’s own vault (45 questions, Sonnet agents, 2026-10-01), kal tools did not prove an advantage: correctness 0.90 with kal vs 0.91 with file tools, with overlapping 95% confidence intervals — which is not the same as a tie. The answer key for that run was written by an LLM. A second LLM judge re-graded 15 of the 45 questions to check that grading agreed; no person re-checked the answers. Questions and paths from that run stay private; only the totals are published.
What these numbers do not say
- That kal is more accurate than grep.
- That it saves context.
- Anything about claude.ai, remote MCP or phones — those clients are agents on strong models, which these runs did not test.