How does memory work in an AI agent, and how is it different from RAG?

The shortest distinction

Memory is what the agent writes down about its own work. Retrieval is what the agent looks up in material somebody else wrote. That single difference in authorship drives almost every other difference between them.

A memory record says the user prefers metric units, or the migration on the orders table failed twice for the same reason. A retrieved passage says what the refund policy is, or what a function signature looks like. The first only exists because the agent ran; the second was true before the agent existed.

Short-term and long-term memory are different systems

LangGraph's documentation splits agent memory by recall scope. Short-term memory is thread-scoped: it tracks the ongoing conversation as part of the agent's state, and that state is persisted to a database through a checkpointer so a thread can be resumed. Long-term memory is stored under custom namespaces, shared across threads, and recallable at any time in any thread.

This split matters because the two have different pressures. Short-term memory fights the context window: the docs note that even models supporting long contexts still perform poorly over them, getting distracted by stale content while costs and latency rise. Long-term memory fights correctness and staleness instead, because a wrong fact written once will be recalled forever.

The same docs borrow a three-way classification from psychology and map it onto agents: semantic memory stores facts about a user, episodic memory stores past agent actions, and procedural memory stores instructions such as the agent's system prompt. It is worth reading that table carefully, because the documentation itself flags a trap: semantic memory is a term about storing facts and is not the same thing as semantic search.

The two shapes vendors actually ship

The first shape is memory as files the model edits. Anthropic's memory tool gives Claude commands to view, create, and edit files under a /memories path, and the model checks that directory before starting a task. The important detail is that the tool is client-side: the model only requests operations, and your application executes each one against storage you control and returns the result. The /memories path is a prefix your handler maps onto real storage, such as a per-user directory or database keys. It is available on Claude 4 and later models.

The second shape is memory as a store the framework manages. LangGraph persists long-term memories as JSON documents under a namespace and key, with cross-namespace search through content filters and optional embedding-based search. Here the framework, not the model, owns the read and write path, and you decide when to write.

There is a third option that is really a packaging decision rather than an architecture: hosted memory-as-a-service. Mem0, for example, markets a drop-in memory layer with an add, learn, retrieve loop, and states it is benchmarked on LoCoMo, LongMemEval, and BEAM. Those are the vendor's own claims and we have not verified them. The tradeoff is the usual one — less code to write, one more third party holding conversation-derived data about your users.

When to write, and why it is the hard part

LangGraph's docs frame the write timing as a real design choice. Memories can be updated on the hot path, meaning the agent decides to record something before it replies, or in a background task that generates memories asynchronously. Hot-path writes are immediately available but add latency and let a bad turn poison the store. Background writes keep the interaction fast but mean the next turn may not see what just happened.

The update pattern matters as much as the timing. A profile is a single continuously updated JSON document, which is simple to reason about but becomes error-prone as it grows, so the docs suggest splitting it or using strict decoding to keep the schema valid. A collection appends discrete records instead, which scales better but pushes the difficulty into retrieval and de-duplication.

Anthropic's docs describe a concrete pattern for multi-session software work that sidesteps a lot of this: an initializer session sets up the memory files deliberately before real work begins — a progress log, a feature checklist, a reference to any startup script — and every later session opens by reading them and closes by updating the log. The stated principle is to work on one feature at a time and mark it complete only after end-to-end verification, not when the code is written. That is a discipline for keeping the log honest, and it is the part most teams skip.

Where retrieval is the right answer instead

Retrieval is a ranking problem over a corpus. OpenAI's retrieval guide describes semantic search over vector stores, where a query returns matching chunks with similarity scores and the file each chunk came from. Their example is a good illustration of what this buys you: for the query when did we go to the moon, the highest-scoring passage is the first lunar landing occurred in July of 1969, which shares no words with the query at all.

That behaviour is exactly what you want for documentation, policies, product catalogues, and codebases — bodies of text that exist independently of any conversation. It is exactly what you do not want for tracking whether step three of a task succeeded.

The decision rule that holds up: if the information would still be true and useful if this agent had never run, it belongs in retrieval. If it only exists because the agent ran, it belongs in memory. Facts about a user sit awkwardly in between, which is why the semantic-memory category exists at all.

Risks that come with the memory, not with the model

Path traversal is a real, documented hazard. Because Anthropic's memory tool is client-side, your handler is the security boundary. The docs are explicit that handlers must restrict all operations to /memories, reject sequences such as ../ and ..\\, watch for URL-encoded traversal like %2e%2e%2f, and use built-in path utilities rather than string checks. If you write the handler yourself, this is not optional hardening — it is the whole control.

Memory also widens the blast radius of prompt injection. Anything the agent reads can influence what it writes to memory, and anything written to memory is read back automatically at the start of later sessions. A single poisoned turn can become a persistent instruction. Treat the memory store as untrusted input on read, not as trusted configuration.

There is a data-handling dimension too. Memory files accumulate whatever the agent found worth recording, which in practice includes user details nobody consciously decided to retain. Because the storage is yours in the client-side model, so are the retention, deletion, and access-control obligations.

Limits of this answer

Everything factual above comes from current vendor and framework documentation checked on 2026-09-01. We have not run benchmarks, and we do not repeat any vendor's performance numbers as fact. The memory benchmarks named on Mem0's page are that vendor's own framing.

The field is moving. Anthropic's memory tool is versioned in the tool type string itself, and both Anthropic and LangChain are actively revising their context and memory guidance. Check the linked documentation before committing to an API shape.

A useful next action

Take one agent that currently forgets things between runs and write down, in one sentence each, what it should remember and why. If every item on that list would still be true had the agent never run, you do not need memory — you need better retrieval, and adding a memory layer will only give you a second place for stale facts to hide.

If items remain, start with the cheapest thing that works: a single append-only progress file the agent reads at the start of a session and updates at the end, following the initializer pattern in Anthropic's docs. Run it for a week before reaching for a namespaced store or a hosted memory service. You will learn what actually needs to persist, which is the input every larger design depends on.