sepiace vs. Claude Code's built-in memory

To show the difference in the setting people actually use, we ran Claude Code with only its built-in memory and Claude Code with sepiace connected, on the same personas and conversations, and asked the same check questions.

Results

Totals for 10 people
MeasureClaude Code DefaultClaude Code + sepiace
Answers and memory
Questions answered correctly75%96%
Facts kept in memory76%99%
Old version left after a fact changed27%5%
Things asked to forget that stayed3%0%
Memory text that matches no fact41%4%
Memory size, relative to the facts3.5×1.3×
KiokuBench Index78.096.1
Tokens and cost
Input tokens to take in 30 sessions (per person)3,326,0003,379,000
Output tokens to take in 30 sessions (per person)66,80040,200
Cost to take in 30 sessions (per person)$5.91$4.00
Input tokens per question18,30050,600

Questions answered correctly, by language

Questions answered correctly, by language
MeasureClaude Code DefaultClaude Code + sepiace
By language
English79%97%
Japanese71%96%

How we compared

  • Claude Code 2.1.296 with Sonnet 5.5.
  • Each person had an empty Claude Code configuration, home folder, and project folder, so no local settings, CLAUDE.md files, memories, or plugins were read.
  • Only the user's turns from the raw conversation were sent, one Claude Code session per conversation session, and Claude Code wrote its own replies. The first message of each session carried that session's date.
  • Built-in memory: Claude Code's auto memory only. Memory measures were taken from its memory files.
  • With sepiace: auto memory turned off, and the sepiace plugin 0.2.1 installed and connected to a local copy of sepiace with the same MCP definition as production.
  • Each check question was asked in a new session started from a copy of the state after the conversations, and GPT-6 Luna judged Claude Code's own answer.

About KiokuBench, the data and scoring we used

Notes

  • Costs are API prices for usage we ran on a subscription. sepiace's own judgment calls cost $0.21 in total.
  • With sepiace, each question reads about 2.8 times as many input tokens, for the plugin's instructions and the MCP tool description. Most of them hit the prompt cache. The cost per question changes a lot with how much hits the cache, so we do not compare it.
  • With plugin 0.2.0, whose instructions only asked to save preferences, decisions, and corrections, Claude Code saved to sepiace 5 to 7 times per person and answered 30% of the questions correctly, on two personas. Version 0.2.1 asks it to save what the user says about themselves and the people in their life.
  • In the first run, a token for the test environment's embedding service expired partway through, and every save failed for the second half of the conversations for two English personas (en-301 and en-302). We fixed the environment and ran those two again. The results here are from the second run.
  • This is a single run on 10 personas.