Read accuracy alone, as in LoCoMo or LongMemEval, rewards systems that store everything. KiokuBench also checks the contents of memory: stale versions, leftovers from forget requests, noise, language, and size.

See KiokuBench v0.1, the older version

KiokuBench v1 r1

KiokuBench v1 is a much harder version than v0.1. Round 1 (r1) is the first round of v1, measured from October 22 to 29, 2026. v0.1 and v1 differ in both questions and method, so their numbers are not comparable.

Aim

We built v1 so that no memory system reaches a perfect score. Fitting to the published questions does not raise the score. It rises only when memory actually gets better. We do not tune scores to any target value, and we publish results where sepiace loses to another system as they are.

How we measure

  • We do not hand conversations to the memory system directly. We connect the memory system to Claude Code or Codex and give the conversations to the agent. The agent decides when to call memory. The benchmark adds no prompt telling the agent how to use it.
  • Each persona has 200 sessions, about 20 months of conversation. About 70% of the sessions are mostly work unrelated to the person. There are 12 personas, 6 in English and 6 in Japanese.
  • Check questions are asked at four points, after sessions 50, 100, 150, and 200, each in a new session. There are 2,916 questions. We dropped 347 from every route's scoring: those that still had a problem generation could not fix, and those answered correctly with no memory. We scored 2,569.
  • Each measurement is a round, and the test set is rebuilt every time. We committed the SHA-256 of the r1 test files before measuring, so anyone can check afterward that they were not swapped after the results came out.
  • KiokuBench Index v1 is a weighted geometric mean of eight pillars. Only the Efficiency pillar has weight 2. It runs from 0 to 100, and higher is better.
  • Claude Code 2.1.311 ran on Sonnet 5.5, and Codex ran codex-cli 0.162.1 with gpt-5.6-terra. Each persona got an empty config and an empty HOME. For sepiace we installed plugin 0.2.1 and swapped only the MCP server for the benchmark's sepiace. In the sepiace routes, we turned off Claude Code's auto memory and Codex's built-in memory. The grader is openai/gpt-6-luna.

Results by memory and agent

KiokuBench Index v1. With both agents, the route with sepiace connected scored highest. Green marks the best value in the same agent column.

KiokuBench v1 r1, Index for 12 personas
MemoryClaude CodeCodex
KiokuBench Index v1
Built-in memory*118.911.4
sepiace60.666.1
No memory12.112.1
  1. *1 Codex creates memories only from interactive sessions, so for its built-in memory alone we drove Codex through its interactive CLI screen, from October 28 to 29, 2026. Codex builds memories in the background after sessions end, so we moved the clock forward and waited for it to finish every 10 sessions and before each check. OpenAI's usage policy check declined to turn some conversations into memories, and we left those conversations out of its memory as Codex did. An earlier run with the non-interactive codex exec created no memories at all.

Results by pillar

Each value runs from 0 to 100, and higher is better. Green marks the best value in the column within the same agent. Efficiency was 100 on every route except Codex's built-in memory.

KiokuBench v1 r1, values by pillar
RouteRememberReplaceForgetDiscernCombineUseHonestyEfficiency
Claude Code
Built-in memory2.44.54.779.40.078.296.3100.0
sepiace51.523.251.280.731.380.189.6100.0
No memory0.00.00.078.00.076.795.0100.0
Codex
Built-in memory9.63.77.071.31.677.592.54.0
sepiace67.931.146.670.148.982.486.0100.0
PillarChecks whether
RememberDetails such as numbers, spellings, and conditions are kept without rewording or mixing up similar things.
ReplaceA change becomes the new fact, and the old version is not kept as current.
ForgetWhat was asked to be forgotten is forgotten, and unrelated things are not erased.
DiscernJokes, hypotheticals, hearsay, and secrets such as passwords are not remembered as facts.
CombineMemories from separate sessions can be linked to trace, count, or order things.
UseMemory is used without being told to, and personal facts are not brought up where they do not belong.
HonestyWhen asked about something never mentioned, it says it does not know instead of making something up.
EfficiencyThe tokens that memory adds per check question and the text written to memory while taking in conversations stay within the limits. This pillar has weight 2.

Fixed-strategy baselines

The benchmark itself ran three fixed strategies for scoring points. Requiring every one to score below 40 is a condition for finalizing a round. In r1, all three scored 22.0 or lower.

KiokuBench v1 r1, fixed-strategy baselines
StrategyIndex
Strategies with no memory system
Always answer that it does not know12.1
Always give some answer15.2
Answer after seeing only the latest 4,000 or so tokens of conversation22.0

What the results show

  • Connecting sepiace raised Claude Code from 18.9 with its built-in memory to 60.6, and Codex from 11.4 with its built-in memory to 66.1.
  • Codex's built-in memory scored 11.4, below no memory at 12.1. It answered 35.8% of the check questions correctly, a little more than no memory at 32.1%. But it wrote about 200,000 tokens of memory per person, about 25 times the Efficiency limit of 8,000, so Efficiency was 4.0.
  • Both are still far from a perfect score. Replace was 23 to 31, and Combine was 31 to 49. Every memory system, sepiace included, has a lot of room to improve.
  • Every fixed-strategy baseline scored 22.0 or lower. Showing only the latest 4,000 or so tokens scored higher than no memory at 12.1, but did not reach sepiace's 60.6 and 66.1.
  • Claude Code + sepiace answered 65%, 61%, 58%, and 59% of the check questions correctly at the four points. Accuracy tends to fall as the conversation gets longer.
  • With sepiace, memory added only 70 to 80 tokens per check question. That is far below the Efficiency limit of 4,000 tokens. With Codex's built-in memory, it added about 2,700 tokens.
  • Discern and Honesty score high even with no memory. A system that remembers nothing never remembers false statements or secrets, and rarely makes up answers about things it does not know. For Claude Code, Honesty was lower with sepiace at 89.6 than with no memory at 95.0. Read these two pillars together with the others.

Limitations

  • KiokuBench was designed and run by the team that builds sepiace.
  • This is a single run on 12 personas. Judging from the variation we estimated on the 10 personas of v0.1, overall scores can shift by about 5 points either way. Do not rank routes whose scores differ by only a few points.
  • The grader is a single model, GPT-6 Luna, and the same model generated the personas and conversations. We have not yet checked its grading against human grading.
  • We have not yet measured Mem0, Supermemory, ByteRover, or TencentDB through the agent route. This round compares only sepiace, each agent's built-in memory, and no memory.