Most teams test their agent's memory by checking whether the fact is stored or retrieved. That's the wrong part to question.

Storage is the easy part. Any key and value store can store a fact. What's hard and what actually determines whether your agent is useful is whether it retrieves the right memory at the right time, uses it correctly in context, and stays silent when it has nothing reliable to say.

A user changes their favorite city from Berlin to Paris. Two weeks later, the agent still recommends Berlin restaurants. Technically, the memory system remembered both facts but practically, it failed.

We spent time researching how the academic community approaches long-term memory evaluation, studying three established frameworks in depth: LoCoMo, LongMemEval, and BEAM. None of them alone was sufficient for what we needed. But together, they gave us a foundation to build something that is.

This is what we learned, and how we combined them.

Why Memory Evaluation Is Hard

The truth about most memory benchmarks is the fact that they measure storage, not behavior.

A system scores well if it can answer "What city does the user live in?" after being told once. But real agent memory is not that simple. Users update facts and contradict themselves. They ask questions that require combining things said in different sessions weeks apart. And sometimes they ask about things they never actually mentioned and the agent should say so.

A single recall score hides all of this. Which is why we went looking for something better.

Frameworks We Studied

LoCoMo — The Reasoning Taxonomy

Full name: Long Context Memory

Core question: Can an AI agent maintain coherent memory across many sessions over time?

LoCoMo's most important contribution isn't a dataset — it's a classification system. It asks “what kind of reasoning does a memory question actually require?” And it breaks down into four types:

This taxonomy exposes something that composite scores hide completely: systems that score 80%+ on single-hop commonly score below 50% on adversarial. An agent that seems to remember well based on average recall can be unreliable when a user updates information or when a wrong answer is available.

LoCoMo also avoids a common trap in memory benchmarks. It uses realistic long-form conversation datasets rather than synthetic Q&A pairs. Questions are asked simulating real user behavior. These questions require understanding information said in completely different sessions and contexts.

For scoring, LoCoMo uses token-level F1 against ground-truth answers. If the correct answer is "Berlin" and the model responds "Berlin, Germany," strict exact-match penalizes it. F1 gives partial credit, which better reflects actual answer quality.

What we took from LoCoMo: The 5-type question taxonomy and added abstention to the original four and F1 as the primary answer quality metric.

LongMemEval — Pipeline Decomposition

Full name: Long Memory Evaluation

Core question: Where exactly does a memory system fail?

This is the framework that changed how we think about memory debugging.

Most evaluation approaches treat memory as a black box. Give an input to a conversation, ask a question, check the answer. If the answer is wrong, you don't know why. Was it a retrieval failure? Did the right chunk get fetched but get ignored in the prompt? Did the model hallucinate on top of correct context?

LongMemEval breaks the pipeline into three independently scored stages:

Stage 1: Store
  "Was the fact saved at all?"
        ↓
Stage 2: Retrieve
  "Was the right chunk fetched?"
        ↓
Stage 3: Use
  "Was the fetched memory used correctly in the response?"

This decomposition is operationally valuable. If retrieval fails, you fix your embedding model or increase k, not your prompt. If the fact was retrieved but the response ignored it, your problem is context window construction. The failure mode points directly to the fix.

The failure diagnosis matrix looks like this:

LongMemEval also makes abstention a first-class metric. Most systems hallucinate when a question was never answered in the conversation. They return a confident response with no grounding. LongMemEval scores this separately with its own precision, recall, and F1, completely independent from answer accuracy. In production, an agent that confidently fabricates is worse than one that gets facts wrong.

LongMemEval uses conversations averaging 115,000 tokens — far beyond what fits in a single context window. This forces systems to actually rely on retrieval rather than attending to raw context, which is how they'll behave in real deployments.

What we took from LongMemEval: The store → retrieve → use pipeline decomposition, and abstention as a first-class metric with its own F1.

BEAM — Degradation at Scale

Full name: Benchmarking Extreme-length Agent Memory

Core question: At what scale does a memory system break?

BEAM asks a different question than the other two frameworks. It is interested in when it stops working. And it varies difficulty across three independent axes simultaneously:

• Context length: how long is the total conversation history? (128K → 10M tokens)
• Noise ratio: what fraction of stored facts are irrelevant distractors? (0% → 90%+)
• Fact distance: how many sessions back was the relevant fact originally injected?

The four difficulty tiers map to real-world deployment scenarios:

‍

The key insight from BEAM is why degradation curves matter more than point scores.

A vector store might score 82% overall. But plotted by tier: 95% at short, 88% at medium, 70% at long, 45% at extreme. That curve tells you something a single number could not: this backend works for short horizon assistants but becomes unreliable for long-term use. And different backends degrade in fundamentally different shapes:

• Vector stores degrade gracefully with fact distance but collapse under high noise ratios
• Structured KV stores hold performance at long distances but drop sharply when noise is introduced
• Summarization backends maintain overall score but progressively lose temporal accuracy as sessions accumulate

Seeing these curves side by side is how you make an informed backend choice not by comparing headline numbers.

What we took from BEAM: The four-tier difficulty structure and three scaling axes — reported as curves, not point scores.

What We Actually Test: 5 Abilities

Combining these three frameworks, we settled on five core abilities that a memory system must demonstrate to be considered production-ready:

Each one maps to a failure mode we've seen in production systems that scored fine on simpler benchmarks.

The Scoring System

Every metric falls into one of three buckets:

Memory (50%) — did the pipeline work?

• Was the fact stored?
• Was the right chunk retrieved?
• Was it used correctly in the response?

Reasoning (30%) — was the answer right?

• Correct answer (F1 vs ground truth)
• Correct time anchor (temporal questions)
• Contradiction surfaced (conflict questions)

Safety (20%) — did it avoid hallucinating?

• Abstention precision and recall (F1)

The composite score is a weighted sum across all three. But we always report the per-bucket breakdown alongside the composite. Two backends can produce an identical composite score with completely different failure profiles. One might have excellent retrieval and terrible abstention. The other might have the inverse. They require entirely different fixes.

Score Patterns Worth Watching

Once you have per-bucket and per-type scores, certain patterns become diagnostically meaningful:

• High retrieval, low temporal → the backend stores facts but doesn't handle updates; old versions of facts survive alongside new ones
• High accuracy, low abstention F1 → hallucination risk; the system answers confidently even when it shouldn't
• High temporal, low multi-hop → structured storage is working, but there's no semantic cross-session search
• High retrieval, low reading → the right chunks are being fetched, but they're not surfaced correctly in the prompt context

These patterns are where the benchmark earns its value.

Three Questions Worth Asking

If you're building any system where an AI agent needs to remember things across sessions, don't evaluate it by asking whether it can store and retrieve a single fact. Ask:

1. Can it reason across sessions? (LoCoMo taxonomy)
2. Where exactly does it fail when it does fail? (LongMemEval pipeline)
3. At what scale does it break, and in what shape? (BEAM degradation curves)

Memory is not a storage problem anymore. It is a reasoning reliability problem. A memory system that scores well on all three is genuinely reliable. A memory system that scores well on only one is telling you something important and it's worth listening.

‍