KiokuBench
KiokuBench measures what a memory system remembered, replaced, and forgot by comparing the stored memories themselves against a ledger of true facts. Kioku is Japanese for memory.
Read accuracy alone, as in LoCoMo or LongMemEval, rewards systems that store everything. KiokuBench also checks the contents of memory: stale versions, leftovers from forget requests, noise, language, and size.
Results for sepiace as of October 14, 2026, and for the others as of September 30, 2026. Supermemory was measured through its API on September 27 and 28, 2026. The open-source Mem0 was run locally on September 29, and ByteRover on September 30, 2026.
We also compared Claude Code's built-in memory with Claude Code + sepiace on the same data
KiokuBench Index
The KiokuBench Index groups the eight metrics into four pillars and combines them into a single number from 0 to 100. Higher is better. The table below is computed from the totals for 10 personas on the route where the raw conversation is passed in as is. Green marks the best value within the same route.
- sepiace96.1sepiaceIndex: 96.1Remember: 98.7Replace: 98.0Forget: 100.0Tidy: 88.3
- Supermemory58.9SupermemoryIndex: 58.9Remember: 95.3Replace: 95.0Forget: 22.0Tidy: 60.5
- ByteRover39.2ByteRoverIndex: 39.2Remember: 72.3Replace: 19.0Forget: 62.0Tidy: 27.7
- Mem08.5Mem0Index: 8.5Remember: 96.3Replace: 0.0Forget: 0.0Tidy: 55.0
- Store everything8.0Store everythingIndex: 8.0Remember: 99.0Replace: 0.0Forget: 0.0Tidy: 41.7
Hover over or focus a bar to see its four pillars. The store-everything baseline is shown in gray for reference.
| System | Index | Remember | Replace | Forget | Tidy |
|---|---|---|---|---|---|
| Raw conversation | |||||
| sepiace | 96.1 | 98.7 | 98.0 | 100.0 | 88.3 |
| Supermemory | 58.9 | 95.3 | 95.0 | 22.0 | 60.5 |
| Mem0 | 8.5 | 96.3 | 0.0 | 0.0 | 55.0 |
| ByteRover | 39.2 | 72.3 | 19.0 | 62.0 | 27.7 |
| Baseline | |||||
| Store everything | 8.0 | 99.0 | 0.0 | 0.0 | 41.7 |
How it is calculated
Each of the eight metrics is first turned into a value from 0 to 1 where higher is better, and then grouped into four pillars.
| Pillar | Made of |
|---|---|
| Remember | Average of recall, update applied, and check questions |
| Replace | 1 − old version kept |
| Forget | 1 − forget violation |
| Tidy | Average of 1 − noise, 1 − language mismatch, and 1 ÷ compression. Compression below 1 counts as 1. |
Pillars below 0.01 are raised to 0.01, and the index is 100 times the geometric mean of the four pillars.
The geometric mean makes the index drop sharply when any single pillar fails. A system that remembers but never forgets, or forgets but never remembers, does not work as memory. The 0.01 floor keeps one zero pillar from hiding the differences in the others.
The values of all eight metrics are in the results tables below.
Results
Totals for all 10 personas. Green marks the better value within the same route.
| Route | Recall | Update applied | Old version kept | Forget violation | Noise | Language mismatch | Check questions | Compression |
|---|---|---|---|---|---|---|---|---|
| Raw conversation | ||||||||
| sepiace | 99% | 100% | 2% | 0% | 12% | 0% | 97% | 1.30 |
| Supermemory | 98% | 100% | 5% | 78% | 20% | 42% | 88% | 2.30 |
| Mem0 | 100% | 100% | 100% | 100% | 47% | 9% | 89% | 4.78 |
| ByteRover | 85% | 95% | 81% | 38% | 70% | 49% | 37% | 49.66 |
| Requests and processing records | ||||||||
| sepiace | 99% | 99% | 13% | 0% | 0% | 0% | 96% | 1.03 |
| Supermemory*1 | 83% | 78% | 35% | 65% | 18% | 42% | 82% | 2.06 |
| Supermemory*2 | 90% | 83% | 46% | 2% | 8% | 42% | 77% | 1.80 |
| Mem0*1 | 100% | 99% | 100% | 100% | 2% | 1% | 88% | 1.79 |
| ByteRover*1 | 93% | 97% | 83% | 25% | 40% | 45% | 34% | 27.92 |
| Change signaled | ||||||||
| sepiace | 99% | 99% | 25% | 0% | 0% | 0% | 94% | 1.15 |
| Supermemory*1 | 92% | 90% | 11% | 40% | 12% | 44% | 89% | 1.95 |
| Mem0*1 | 100% | 100% | 100% | 100% | 2% | 2% | 91% | 1.80 |
| ByteRover*1 | 96% | 99% | 91% | 20% | 38% | 39% | 33% | 28.22 |
| Baseline | ||||||||
| Store everything | 100% | 100% | 100% | 100% | 83% | 0% | 97% | 12.30 |
- *1 Without routing. Forget requests are sent like any other request, and the system decides by itself. Same condition as sepiace.
- *2 With routing. Only forget requests are sent to Supermemory's forget-matching endpoint.
The store-everything baseline keeps every utterance as is. For check questions, it passes the utterances closest in embedding up to the context limit.
By language
| Route | Recall | Update applied | Old version kept | Forget violation | Noise | Language mismatch | Check questions |
|---|---|---|---|---|---|---|---|
| Raw conversation (English) | |||||||
| sepiace | 100% | 100% | 2% | 0% | 9% | 0% | 98% |
| Supermemory | 100% | 100% | 4% | 70% | 15% | 0% | 89% |
| Mem0 | 100% | 100% | 100% | 100% | 44% | 0% | 90% |
| ByteRover | 83% | 96% | 74% | 35% | 69% | 0% | 61% |
| Raw conversation (Japanese) | |||||||
| sepiace | 98% | 100% | 2% | 0% | 15% | 0% | 96% |
| Supermemory | 97% | 100% | 6% | 85% | 25% | 79% | 87% |
| Mem0 | 100% | 100% | 100% | 100% | 50% | 17% | 87% |
| ByteRover | 87% | 94% | 88% | 40% | 70% | 87% | 13% |
| Requests and processing records (English) | |||||||
| sepiace | 99% | 98% | 14% | 0% | 0% | 0% | 94% |
| Supermemory*1 | 68% | 58% | 30% | 60% | 30% | 0% | 86% |
| Mem0*1 | 99% | 98% | 100% | 100% | 2% | 0% | 88% |
| ByteRover*1 | 97% | 100% | 82% | 20% | 36% | 0% | 54% |
| Requests and processing records (Japanese) | |||||||
| sepiace | 100% | 100% | 12% | 0% | 1% | 0% | 97% |
| Supermemory*1 | 98% | 98% | 40% | 70% | 7% | 81% | 77% |
| Mem0*1 | 100% | 100% | 100% | 100% | 3% | 3% | 88% |
| ByteRover*1 | 90% | 94% | 84% | 30% | 43% | 80% | 14% |
| Change signaled (English) | |||||||
| sepiace | 99% | 100% | 10% | 0% | 0% | 0% | 95% |
| Supermemory*1 | 87% | 84% | 2% | 15% | 20% | 0% | 95% |
| Mem0*1 | 100% | 100% | 100% | 100% | 0% | 0% | 91% |
| ByteRover*1 | 94% | 100% | 90% | 10% | 35% | 0% | 51% |
| Change signaled (Japanese) | |||||||
| sepiace | 99% | 98% | 40% | 0% | 1% | 0% | 93% |
| Supermemory*1 | 98% | 96% | 20% | 65% | 5% | 84% | 83% |
| Mem0*1 | 100% | 100% | 100% | 100% | 3% | 5% | 92% |
| ByteRover*1 | 97% | 98% | 92% | 30% | 41% | 77% | 15% |
What the results show
- Across all three routes, sepiace had the lowest forget violations, noise, and language mismatch, and the best check-question scores. The exception is old version kept on the change-signaled route, where Supermemory was lower.
- On that route, sepiace kept the old version 25% of the time, and 40% in Japanese. In an earlier run, 17 of 22 such cases were facts where only part changed. For example, “I swim at the city pool every Tuesday at 6 p.m.” followed by “I moved my swim from 6 to 6:30.” The new sentence mentions only the time, so removing the old one would lose the pool. Since sepiace never rewrites memory text, it keeps both.
- Mem0 kept every old version and every forgotten item: both were 100% on every route. This is how Mem0 2.2.1 is designed. Each add only appends new memories and never rewrites or deletes existing ones, so recall is near 100% because nothing is ever dropped. A forget request itself became a new memory that restates what was to be forgotten.
- ByteRover does rewrite and delete memories, unlike Mem0, but forget violations still remained at 20–38%, and 81–91% of changed facts kept their old version in the file's change log. It stored Japanese conversations almost entirely in English, and its word-based search found nothing for most Japanese check questions.
- Supermemory stored about 80% of Japanese input as English memories.
- Ingestion cost with the same LLM (openai/gpt-6-luna): ByteRover made 9,034 LLM calls for $8.11, about 6.6 calls per memory request. Mem0 made 1,377 calls for $0.36.
Metrics
| Metric | Meaning | Better |
|---|---|---|
| Recall | Share of facts true at the end that remain in memory | Higher |
| Update applied | Share of changed facts whose new version is in memory | Higher |
| Old version kept | Share of changed facts whose old version also remains in memory | Lower |
| Forget violation | Share of content the user asked to forget that still remains | Lower |
| Noise | Share of memory sentences that match no fact in the ledger | Lower |
| Language mismatch | Share of memory sentences written in a different language from the conversation | Lower |
| Check questions | Share of check questions answered correctly from about 1,000 tokens of context built from memory | Higher |
| Compression | Characters in memory divided by characters in the ledger facts. Closer to 1 means less waste | Closer to 1 |
Routes
There are three ways to feed data in.
- Raw conversation. The conversation is passed in as is, and the memory system decides what to keep. sepiace receives one utterance at a time. Supermemory receives each session as one document. Mem0 receives each session's messages in one add call, and ByteRover receives each session in one curate call.
- Requests. “Remember” and “forget” requests built from the ledger are passed in. For a changed fact, only the new fact is sent as “remember”, without saying that it changed.
- Change signaled. The same as Requests, but a change request says that the fact changed from A to B.
On the request routes, forget requests are sent like any other request, and each system decides by itself. This is the same condition for every system. For Supermemory, a second setup was also tried: only forget requests are sent to its forget-matching endpoint, which tells it the kind of operation.
Dataset v0.1
- 10 personas: 5 in English and 5 in Japanese.
- Each persona has a ledger of 40 facts, or 39 for two personas, with 30 conversation sessions built from the ledger and one check question per fact.
- 10 facts change along the way, and the user asks to forget 4 of them.
- The ledgers, conversations, and questions were generated with openai/gpt-6-luna on OpenRouter.
- A grader model checked that the conversations follow the ledger. It found one fact not in the ledger and two requests to forget only part of a fact. No human review has been done yet.
Conditions
- The grader and the answerer are both openai/gpt-6-luna on OpenRouter.
- sepiace was measured with its code as of October 14, 2026.
- Supermemory ingested with dreaming set to instant. The default, dynamic, builds memories about 10 minutes after ingestion pauses, and waiting for that after every conversation would take more than 5 hours per persona.
- Supermemory's forget-matching was used with its default settings.
- Supermemory's search used the settings Supermemory itself uses in MemoryBench: hybrid, threshold 0.3, 30 results.
- Mem0 is the open-source mem0ai 2.2.1, run locally. Its internal LLM is openai/gpt-6-luna, its embedding model is BAAI/bge-m3 (the same as sepiace), and it stores memories in a local Qdrant. It was installed without optional extras, so its search uses embeddings only. Search returns 30 results with Mem0's default threshold of 0.1.
- ByteRover is byterover-cli 3.16.1 (brv, Elastic License 2.0), run locally without a ByteRover account, cloud sync, or hosted LLM. Its LLM is openai/gpt-6-luna. Each input starts with its date, since curate has no date argument. Its clean-up (dream) was run once per persona after ingestion. Check questions use brv query, fitted to the same context budget as the other systems.
Limitations
- KiokuBench was designed and run by the team that builds sepiace. Its metrics reward what sepiace was built to do, such as forgetting on request and keeping the language of the conversation, so read the results with that in mind. The other systems were run with the settings listed under Conditions.
- This is a single run on 10 personas. Judgments vary, so the same code can shift by a few cases from run to run.
- The dataset is small and has not been reviewed by people yet. A larger v1 is planned.