Why three layers

A long-running agent needs memory that is fast, durable, and searchable. No single storage mechanism is good at all three. That's why real systems end up with three layers, each doing one thing well.

The mistake most first attempts make is picking one layer and trying to make it do all three jobs. That is how you end up with a gigantic session transcript that the agent can't search, or a beautiful vector store with nothing useful in it.

Layer 1: Short-term memory

Short-term memory is the current conversation. It lives inside the model's context window and disappears when the session ends.

What goes in: the user's messages, the agent's responses, the tool calls and results.

What comes out: the answer to the next prompt.

Rules of thumb:

  • Cheap. No storage cost.
  • Fast. No retrieval cost.
  • Bounded. Limited by the model's context window.
  • Volatile. Gone when the session ends.

Most of what the agent "knows" at any given moment is short-term memory. That's a feature, not a bug — short-term memory is the freshest possible information. The mistake is assuming short-term memory is enough.

Layer 2: Long-term memory

Long-term memory is the notes the agent has saved about this user, project, or run. It usually lives as a markdown file in the workspace, sometimes split across multiple files.

What goes in: stable facts, decisions, open threads, anything the agent will need to find next session.

What comes out: the context that gets loaded into the prompt at the start of a new session.

Rules of thumb:

  • Curated, not raw. Long-term memory is the agent's interpretation

of what matters, not a transcript of what happened.

  • Editable. A human should be able to read, change, or delete any

entry without a tool call.

  • Smaller than the conversation that produced it. If the memory

file is longer than the chat that wrote it, you're keeping too much.

  • Read on intake. The simplest pattern is "load the memory file at

the start of every session."

The single biggest mistake in agent work is treating long-term memory as a database. It's a notebook. Notebooks are good at being read by humans. That's the feature.

Layer 3: Semantic memory

Semantic memory is indexed snippets retrieved by relevance. It usually lives in a vector store and is queried at context-assembly time.

What goes in: chunks of conversation, documents, or notes, converted into embeddings and stored.

What comes out: the top-K most-relevant chunks for the current prompt.

Rules of thumb:

  • Useful when you have more material than you can fit in the

context window.

  • A retrieval aid, not a memory replacement. The retrieved chunks

are inputs to the model's reasoning, not decisions.

  • Quality of retrieval is everything. Bad embeddings, bad chunks,

bad prompts — all silently degrade the agent's "memory" without throwing an error.

  • Not a substitute for long-term memory. If you find yourself

relying on semantic retrieval for facts the agent should obviously know, the agent's long-term memory is the actual bug.

How the three layers fit together

At intake, the agent assembles context roughly in this order:

1. System prompt. Identity, role, capabilities. 2. Long-term memory. Saved notes about the user, project, prior work. 3. Semantic retrieval. Top-K snippets from the vector store, selected by the current prompt. 4. Recent session transcript. The last N turns of conversation.

That four-part prompt is what the model sees. The model's output goes back into the conversation. The agent may also write to long-term memory at the end of the loop.

This is the boring right answer. It's also what most production agents do, regardless of platform or framework.

What goes wrong at each layer

Short-term memory fails when the context window overflows. The first thing the model "forgets" is the middle of the conversation, not the beginning or the end. Keep important context in long-term memory, not just in chat.

Long-term memory fails when the agent saves too much, or the wrong things, or in a place the human can't edit. The agent becomes over-confident in its own notes. The operator loses visibility. The fix is almost always to delete things, not to save more.

Semantic memory fails when the chunking is wrong, the embeddings are stale, or the retrieval step is gating too tightly. Symptoms look like "the agent knows things one day and forgets them the next." That is rarely a long-term memory problem — it's almost always a retrieval problem.

A field note for next time

When the agent seems to have forgotten something:

  • Check short-term memory first. Did the conversation get long

enough to push the fact out of the context window?

  • Check long-term memory second. Was the fact saved in the first

place? Is it still in the memory file?

  • Check semantic memory last. Is the fact retrievable? Is the

embedding model still the one the rest of the system expects?

The order matters. Most "the agent forgot" stories are short-term memory overflow, not memory-system failure. The fix for that is often to write the fact to long-term memory, not to improve retrieval.