Read accuracy alone, as in LoCoMo or LongMemEval, rewards systems that store everything. KiokuBench also checks the contents of memory: stale versions, leftovers from forget requests, noise, language, and size.

See every version of KiokuBench

KiokuBench v2.0 r1

KiokuBench v2.0 keeps the questions of v1 and measures other memory systems too, at a cost that lets us run it after every change. Round 1 (r1) was measured on November 5, 2026. sepiace was measured again on November 6, after we fixed what r1's dev set showed it was missing. The other five routes were not measured again. Do not compare v2.0 numbers with v1.0 or v0.1.

How we measure

  • Every memory system is connected to Claude Code 2.1.318 running Haiku 5.5. The agent decides when to call memory, and the benchmark adds no prompt telling it how. Each route allows only its own memory tools.
  • There are 12 personas, 6 in English and 6 in Japanese, each with 200 sessions. Check questions are asked after sessions 50, 100, 150, and 200, each in a new session. There are 2,849 questions; we dropped 413 that still had a problem generation could not fix and scored 2,436.
  • Answers are scored by matching them against the correct values, with no LLM grader, so the same answer always gets the same score. Memory contents are checked the same way for secrets, forgotten content, untrue claims, and stale versions.
  • We committed the SHA-256 of the r1 test files before measuring, so anyone can check that they were not swapped after the results came out.
  • The Index uses the same formula as v1: a weighted geometric mean of eight pillars, with the Efficiency pillar at weight 2. We plan to revise the Efficiency pillar for v2.
  • sepiace was measured with plugin 0.3.0 and Jev 1.14, after the fixes in r1's dev analysis. Before the fixes, with plugin 0.2.1 and Jev 1.13, it scored 51.2.

How each memory was connected

We ran every other memory system ourselves from its self-hosted version, with the LLM inside each one set to openai/gpt-6-luna.

MemoryHow it was connected
sepiaceThe public plugin (osaxyz/sepiace) with only the MCP server swapped for the benchmark's. Plugin 0.3.0.
TencentDBClaude Code's API endpoint pointed at the proxy of self-hosted TencentDB Agent Memory 2.0.3.
Built-in memoryClaude Code's auto memory.
Mem0The official Claude Code plugin (0.3.3), pointed at Mem0 OSS 2.2.1 running locally.
ByteRoverThe self-hosted CLI (byterover-cli 3.16.1) and the skill that ByteRover recommends for Claude Code.
ObeliskThe official CLI (@obelisk-apps/cli 0.3.1) and skill.

Results by memory

KiokuBench Index and the share of check questions answered correctly. Green marks the best value in each column.

KiokuBench v2.0 r1, 12 personas, Claude Code on Haiku 5.5
MemoryIndexEnglishJapaneseCorrect
KiokuBench Index
sepiace*154.757.751.467.5%
TencentDB47.947.748.059.2%
Built-in memory19.416.021.030.5%
Mem018.818.518.564.1%
ByteRover14.314.314.629.4%
Obelisk5.25.74.431.3%
  1. *1 Measured again after fixes. Before the fixes, sepiace scored 51.2 with 53.9% correct.

Scores by pillar

From 0 to 100, higher is better. Green marks the best value in each column.

KiokuBench v2.0 r1, scores by pillar
MemoryRememberReplaceForgetDiscernCombineUseHonestyEfficiency
Pillars
sepiace78.46.345.662.347.968.095.6100.0
TencentDB67.27.617.567.039.366.296.993.8
Built-in memory6.32.25.874.10.565.899.1100.0
Mem080.11.72.841.347.468.795.47.7
ByteRover5.40.80.072.90.565.098.5100.0
Obelisk11.40.01.80.04.365.398.32.3

Cost

Agent cost only, for all 12 personas. It does not include the LLMs inside each memory system or sepiace's Jev. Green marks the lowest cost.

KiokuBench v2.0 r1, agent cost
MemoryTotalPer questionMemory tokens per question
Cost
sepiace$32.93$0.0015127
TencentDB$50.36$0.0033Not measurable
Built-in memory$29.24$0.001892
Mem0$28.39$0.0011275
ByteRover$26.87$0.001315
Obelisk$28.39$0.0017381

Fixed-strategy baselines

Every baseline must stay below an Index of 40 for the round to count. In r1, all three did.

KiokuBench v2.0 r1, fixed-strategy baselines
StrategyIndex
Strategies that use no memory system
Always answer that it does not know11.9
Always give some answer14.3
Answer after seeing only the latest 4,000 or so tokens of conversation24.5

What the results show

  • sepiace ranked first in both the Index (54.7) and correct answers (67.5%), ahead of Mem0 (64.1%) and TencentDB (59.2%), at less than half of TencentDB's cost per question.
  • Before the fixes, sepiace was first on the Index but behind Mem0 and TencentDB on correct answers (53.9%). Most of what it missed were facts the agent never asked it to save. After the fixes, questions answered with "I don't know" fell from 657 to 257.
  • sepiace dropped on Forget (51.9 to 45.6) and Discern (71.6 to 62.3). Saving more meant saving jokes and hypotheticals too, and it still removes facts that should survive a forget request. This is our next fix.
  • Mem0 scored highest on Remember (80.1) but only 18.8 on the Index. It wrote about 100,000 tokens of memory per person, which pushed Efficiency down to 7.7, and 92% of what users asked it to forget was still in memory.
  • TencentDB cost about 1.7 times as much as the other routes, because its proxy adds memory to the prompt on every message.
  • With Haiku 5.5, Claude Code's built-in memory and ByteRover stored almost nothing. Both scored below the baseline that only sees the most recent conversation (24.5).
  • Obelisk keeps whole transcripts, so it keeps every secret too. That put Discern at 0 and Efficiency at 2.3.
  • No memory scored above 10 on Replace. None can yet reliably answer with the current value of a fact that changed.

Limits

  • Only Claude Code on Haiku 5.5. A stronger model decides differently when to call memory, which could change the results, especially for the built-in memory and ByteRover.
  • Matching by string can count a correctly reworded answer as wrong. On v1 r1's answers, it gave the same score as the LLM grader 94.1% of the time.
  • Memory contents are matched by string, which is coarser than the LLM mapping v1 used.
  • We ran the other memory systems from their self-hosted versions and changed some settings for the benchmark. Their cloud versions may score differently. The details are in our design notes.
  • Supermemory has not been measured yet. We will measure it under the same conditions and measure sepiace again on the same day.