---
title: "Agent memory needs a chain of custody"
date: "2026-07-07"
description: "Vector stores answer 'what did we save about X'. Production needs 'why should the agent believe it'. What forensics, W3C PROV, and Certificate Transparency already know about evidence, and how I applied it to the memory layer my own agents run on."
category: "AI"
readTime: "11 min read"
tags: ["agents", "memory", "provenance", "rust", "production-ai"]
---
This morning my agent's memory system opened the session with this line:

```
Store trust: WARN · verify before use (score 0.50)
```

That is the memory layer telling the agent not to trust the memory layer. It computed
that verdict itself, from its own health signals, and every recall in that session
carried the warning with it. I built it to do that, and this essay is about why that
line is the most important thing a memory system has ever told my agents.

The claim I want to defend: an agent memory that cannot say where each item came from,
when it was true, and what supports it is not memory. It is a junk drawer with
embeddings, and agents act on junk with a straight face.

## The junk drawer

The standard shape of agent memory in 2026 is still often a vector store with a nicer
API. Embed the conversation, retrieve by similarity, stuff the context window. Many
frameworks ship a variant of it, and that shape answers exactly one question: what did
we save that looks like this query?

In production that turns out to be the easy question. The hard question arrives around
week three, when the store has accumulated enough state to contradict itself, and it
has four reliable ways of hurting you:

**Staleness.** A fact was true when written and the world moved. Someone's role
changed. A dependency was replaced. A pricing decision was reversed. Similarity search
has no concept of "was true"; the stale version embeds just as well as the current one,
and often better, because it was written more confidently.

**Reversal.** A later decision supersedes an earlier one, and both are in the store.
Retrieval returns whichever is closer to the query phrasing. I have watched an agent
argue for an architecture we had explicitly abandoned, citing my own notes at me,
because the abandonment memo used different vocabulary than the original design.

**Contradiction.** One entry says a person is VP of Engineering. A later one says CTO.
A memory layer that averages, ranks, or silently picks one is an agent that writes a
confidently wrong email to a real person. The conflict itself is the information, and
the standard stack has nowhere to put it.

**Poisoning.** Memory is now an attack surface, not just a quality problem. If
instruction-bearing content from a document, a web page, or another agent gets written
into long-term memory, it executes later with the trust of a recalled fact. OWASP's
agentic security Top 10 tracks this as
[ASI06, memory poisoning](https://owasp.org/www-project-agent-memory-guard/), and it is
the failure mode that turns all the others from bugs into vulnerabilities.

None of these are retrieval problems. Better embeddings make them worse, because
retrieval gets more convincing while the underlying epistemology stays empty. The
missing piece is not recall quality. It is believability, and believability is a
well-developed discipline in fields that have had to take evidence seriously for much
longer than agent frameworks have existed.

## The field is solving a different problem

To be fair to the ecosystem: agent memory is a crowded, fast-moving space, and the
serious players are doing real work. It is worth being precise about which question
each of them answers.

[Letta](https://www.letta.com/), descended from the
[MemGPT paper](https://arxiv.org/abs/2310.08560), treats memory as an
operating-systems problem: paged context, self-editing memory blocks, an agent that
manages what stays resident. That answers "how do I fit a long life into a short
context window", and it answers it well.

[Zep](https://www.getzep.com/), built on
[Graphiti](https://github.com/getzep/graphiti), is the most serious attempt at the
staleness problem: facts carry validity intervals, and Graphiti's public docs make
episode provenance explicit, so the graph can represent that something was true,
stopped being true, and came from a raw episode. That answers "when was this true" and
"where did this derived fact come from", both genuinely part of custody.

[Mem0](https://arxiv.org/abs/2504.19413) sits at the extraction layer: deciding what
is worth remembering from a conversation, consolidating duplicates, updating what
changed. That answers "what should be remembered at all".

Capacity, temporality, salience. All three make recall smarter, and if my problem were
recall quality I would use them rather than compete with them. The remaining gap is on
the caller side of the contract. When a memory item is handed to an acting agent, I want
an explicit verdict on that item, a store-level authority decision for the current
projection, and a history that can prove later what the store said. Graphiti gets
closer than a flat vector store by carrying temporal facts and episode provenance, but
the public memory stack still largely optimizes the librarian. The question I care
about is auditing the library.

That question, "why should the agent believe this", is the one nahuali exists to
answer, and it is deliberately not competing on the other three.

## Evidence has rules

Forensics solved this problem before computers existed. Physical evidence is admissible
in court only with a documented chain of custody: who collected it, when, where it was
stored, who touched it, and whether it changed. Break the chain and the evidence can be
thrown out regardless of whether it is true, because nobody can distinguish true from
tampered anymore. NIST wrote the digital-forensics version down two decades ago in
[SP 800-86](https://csrc.nist.gov/publications/detail/sp/800-86/final): preserve the
integrity of the information, keep a strict chain of custody, and treat evidence
handling as part of the incident-response process rather than a footnote.

Data systems met the same problem and standardized an answer. The W3C
[PROV data model](https://www.w3.org/TR/prov-dm/) describes any piece of data through
three primitives: the entity itself, the activity that produced it, and the agent
responsible for that activity. It exists because "what does the table say" and "why
does the table say it" are different questions, and lineage is the only bridge between
them.

And the web's certificate infrastructure learned the hardest version of the lesson.
Certificate authorities occasionally issued certificates they should not have, and for
years nobody could tell, because a CA's log was whatever the CA said it was.
[Certificate Transparency](https://datatracker.ietf.org/doc/html/rfc6962) fixed the
observability half of that problem with append-only, cryptographically verifiable logs:
misissuance still happens, but it became detectable, because the log itself can no
longer be silently rewritten.

Three fields, one conclusion: a record you cannot trace and cannot prove intact is not
evidence. Too much agent memory still ships with neither property in the part the
caller actually sees.

## What custody means for a memory store

[Nahuali](https://github.com/Arakiss/nahuali) is my attempt to build the memory layer
those three disciplines would design. It is local-first, source-available, written in
Rust, and its design reduces to five mechanisms. None of them is exotic. The work is
deciding that memory deserves them.

**Provenance on assertional memory.** Facts, links, procedures, and intentions can
carry a `source_episode_id`: what happened, when, in which project context. This is
PROV's entity-activity-agent habit applied to agent state. A fact without a source
episode is still storable, but it is not allowed to look identical to a supported fact:
health inspection and recall trust mark it as unsupported, and that mark travels with
the result.

**Verdicts at recall time, not write time.** When my agents query the store, results
come back qualified. An evidence-backed claim or link arrives as `CERTIFY` with a score
of 1.00 and cites the episode it rests on. An unsupported claim degrades to `WARN`; an
unsupported procedure or intention is only `ADVISORY`; a high-risk local signal can
`BLOCK` trust for that result. The consumer sees the epistemic status next to the
content, the way a lab report shows the confidence interval next to the measurement.

**Health as first-class state.** The store can inspect projected memory through named
dimensions: completeness, confidence, freshness, connectivity, contradiction,
staleness, unsupported memory, and blind spot. The API exposes counts, structured
signals, severity, evidence IDs, and a store-level authority decision. The `WARN` at
the top of this essay was not a hardcoded disclaimer. It was computed from real
signals, and it changes agent behavior: under `WARN`, my agents treat every recall as a
lead to verify against the repository, never as an authority to act on.

**Contradictions surfaced, never averaged.** If the store holds both "VP of
Engineering" and "CTO" for the same person, that conflict is visible state attached to
the entity, and recalls that touch it say so. Resolving the conflict is the operator's
job. Hiding it is the one thing the system refuses to do.

**A tamper-evident history.** The CLI, MCP server, and local API hash-chain recorded
events by default; the core crate keeps that behavior behind the `tamper-evidence`
feature. Ed25519 tip attestation is opt-in. This is the Certificate Transparency move
applied to an agent's past: nobody, including me, including a compromised agent, can
silently rewrite what the store previously said once the tip is anchored. The codebase
includes a one-command demonstration (`nahuali demo` in an `attestation` build) that
builds a small ledger in memory, tampers with it, and shows the detection, because a
tamper-evidence claim you cannot re-run yourself is exactly the kind of claim this
essay argues against.

One honest limit belongs right here: a ledger is lie-preserving. If you sign a rumor,
you own a durable, cryptographically intact rumor. Tamper evidence proves the history
was not rewritten. It says nothing about whether the history was true when written.
That is why the ledger is the last mechanism on the list and provenance is the first;
integrity without lineage just fossilizes your mistakes.

## The consumer side of the contract

Custody constrains the reader too. My agents operate under a standing rule: recalled
memory is evidence, not permission. A `CERTIFY` verdict lowers the verification burden:
the agent may use it as supported context, but public, costly, or irreversible actions
still need fresh ground truth. `ADVISORY`, `WARN`, and `BLOCK` force uncertainty,
re-verification, or escalation before action.

This costs real money. Re-verification burns tokens that a trusting agent would save,
every session, and most of the time the memory was fine. I pay that tax deliberately,
for the same reason finance teams pay for reconciliation: the wrong action taken
confidently from stale state costs more than years of verification overhead. The
incident that fixed this rule in place was small and embarrassing. An agent quoted a
project decision back to me as current when it had been reversed weeks earlier, and the
only reason it caused no damage is that I happened to be watching. The tax is what
"happened to be watching" costs when you automate it.

There is a second-order effect I did not design for: knowing that every fact carries
provenance changed what gets written. Vague summaries make bad evidence, so the agents
learned to record specifics, names, dates, and outcomes, because that is what survives
a verdict check. Custody discipline turned out to be a forcing function for memory
quality at write time. Evidence rules improve the witnesses.

## Regulation is arriving at the same place

I did not build this because a regulator asked. But it is worth noticing where the
regulators are heading. The EU AI Act's
[Article 12](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-12) requires
high-risk AI systems to automatically record events over their lifetime, with enough
traceability to monitor operation and support post-market review. OWASP now maintains an agentic
security track where memory integrity is a named threat category. The direction is not
subtle: for systems where autonomy and risk meet, logs and traceability are becoming
part of the product contract.

Teams shipping agents on junk-drawer memory will retrofit custody under audit pressure,
on a deadline, after an incident. It is cheaper to design for it. It was cheaper even
for one person.

## What I do not have

The honest scorecard, because this blog has a policy about claims. Nahuali is early and
small: two GitHub stars, one operator, and its heaviest users are my own agents. The
trust scores are deterministic thresholds checked by fixtures, not learned weights
validated on anyone else's workload. Recall logging captures what was retrieved, which
means I can audit what the agent saw but still cannot measure what it should have seen
and missed; a Nahuali-specific recall-quality eval does not exist yet. And the whole
design assumes a local-first, single-tenant world, which conveniently sidesteps the
hardest multi-tenant custody questions.

What I do have is the property this essay is actually about: when my memory system is
degraded, it says so, to the agent, at recall time, with a score and a reason. Every
item can answer where it came from. The history can prove it was not rewritten.

## The question to ask your memory layer

If you run agents in production, pull ten items out of your memory store and ask each
one: could this testify? Where did it come from, when was it true, what supports it,
and can you prove nobody edited it after the fact?

If the answers are embeddings, you do not have memory. You have a junk drawer, and the
agents rummaging in it are making decisions on your behalf.