How to benchmark agent memory

Test the four operations that define memory

• Updates. Tell the system a fact, change it later, then ask for the current value. The test is whether the new value replaces the old one, not whether the old one can be found.
• Identity. One person appears as an email signature, a name in a transcript and a chat handle. The test is whether the system treats them as one person across sources.
• Historical questions. Ask for a value as it stood on a past date. The test is whether the system keeps when each fact was true, and can return the right one for that moment.
• Unsupported questions. Ask what the history cannot answer. The test is whether the system says so instead of inventing support.

Public suites cover parts of this. LongMemEval tests updates and abstention, LoCoMo tests recall across long conversations, and BEAM tests ten abilities at history sizes up to 10M tokens. Run several; no single suite covers all four operations.

Required baselines

Use two baselines. The full-context baseline sends the whole history with every question and shows accuracy and cost with no retrieval at all. The plain-RAG baseline retrieves passages from the same history. Running both shows what the memory system adds over simpler methods.

Separate retrieval and answer scores

Retrieval measures whether the returned evidence contains the fact needed, and whether current facts rank above the ones they replaced. Answer accuracy measures the final response from the model that reads that evidence. Report the two as separate numbers. Use more than one judge, publish the judge prompts, and report each judge’s results, because the choice of judge moves scores.

The publication checklist

1. Dataset, split and answer-key version, with any corrections named.
2. System version and full configuration: retrieval limits, token budgets, models.
3. History size and question counts.
4. Answer model and every judge, with prompts.
5. Accuracy, cost and latency from the same run, by category and in aggregate.
6. Failed, timed-out and degraded requests, counted, not dropped.
7. The harness, runnable by someone other than the publisher.

Harny’s results are on the benchmarks page.

How Harny’s token and cost figures were measured

98% fewer tokens, 50× less to read. On 7 October 2026, one fictional customer with 163 records (80 emails, 58 calendar events, 25 notes) was asked 12 everyday questions about family, travel, work and money. For each question we counted the tokens the agent’s model receives, with the tiktoken cl100k_base tokenizer. Harny’s context ran 13,779 tokens over the 12 questions (median 1,105 per question). The raw output of the three tools ran 783,324 tokens (median 65,277): 56.9× fewer, which is 98%. Handing the agent only the one tool that holds the answer ran 404,872 tokens: 29.4× fewer. The same day we fixed how the question reader handles a recurring weekday, and Harny’s context then answered 12 of 12 correctly, as both raw baselines did (13,874 tokens, 56.5×). One model (glm-5.3-flash) answered from each context; a second (deepseek-flash) graded each answer against a reference written before the run.

4× cheaper per answer. On 15 September 2026, 125 LoCoMo questions were answered four ways. A cheap model (deepseek-flash) on Harny’s 4,096-token context cost $1.81 per 1,000 answers at list prices. A mid-tier model (gpt-5-mini) reading the whole history cost $7.16: 3.96× more. Strict grading (deepseek-v4-pro) gave 78.9% against 77.6% for the whole history, and the answer arrived in 3.3 seconds against 17.2.