Rebuilding Libra’s memory as a system

Finding out that a large part of the memory we were building inside Libra was an uncomfortable find. What started as a regular testing and checking of the memory feature led to deeper research on how Libra was learning, storing, indexing, and recalling.

What we found was that recall ran on every message: it searched memory, ranked what came back, and returned a result the assistant never saw. Separately, a "knowledge graph" was written on every update and read by nothing downstream. We had a system that learned and stored diligently, and then quietly talked to itself.

That was the moment we stopped treating memory as a collection of features and started treating it as one system to design.

How it got that way

None of this came from a single bad decision. Libra's memory grew the way most systems do: each feature that needed to remember something brought its own way of remembering it. The email assistant stored things one way, meetings another, documents a third. Every addition was reasonable on its own. Together they formed a memory that no one had designed as a whole.

The failures followed from that. Every new message re-processed the entire conversation from the beginning, redoing work that hadn't changed. Old facts and new facts sat next to each other with nothing marking which one was currently true. And workspace memory was stored message by message, with edits and deletes often missed. As a result something you had deleted could still surface in a later search.

That last failure mattered most to us. A memory that can't reliably be forgotten isn't just inefficient. It's a trust problem.

What we measured first

Before rebuilding anything, we wanted to know why recall kept failing to find things that were plainly in memory.

So we tested it directly. For example, we took a short message - a single line answering a specific question and stored it on its own. Later, we asked the exact question that line answers but it rarely comes back. What it meant was on its own, a short message doesn't carry enough surrounding context to be findable, even by the exact question it was written to answer.

So then we stored the same answer together with the discussion around it, and asked again. This time it surfaced first.

That result shaped everything after it because the unit of memory was wrong. We had been storing fragments, and fragments don't remember well. So we changed what a memory is: whole conversations and whole documents , not isolated lines.

What we rebuilt

From there, most of the work was making memory behave like one designed system.

One current version of each fact: Every fact now lives in one place, and only one version of it can be marked current, now the storage layer enforces this rather than trusting each feature to get it right. Older versions aren't discarded; they're kept as history and linked to whatever replaced them. When the assistant recalls a fact, it gets the true one, and can still trace what it used to be.

Only process what's new: Instead of a flag that marks a message "seen or not seen," we keep a bookmark, a record of how far into a conversation we've already read. When new messages arrive, we resume from the bookmark rather than re-reading everything before it. The cost of remembering a conversation no longer grows with its length.

Deletes reach memory: When you delete something, the deletion now propagates into memory itself, so deleted things stop being findable. This is the fix we cared about most, because it's the one a person can actually feel and the one most tied to whether they can trust the product with their work.

Relationships without a graph database: We wanted facts to connect to each other without running a separate graph database. Each record points to the ones it's related to, and recall follows those links one step. That gives us most of what a graph would, without another system to maintain.

One index for everything. Facts, conversations, and documents are now searched and ranked together in a single index, rather than living in separate stores that each answer in their own way. A query sees all of memory at once.

Meaning-based search, with a fallback. Recall works primarily by meaning, using vector search, so a question can find an answer that means the same thing even when the wording is different. If that path is unavailable, it falls back to keyword search instead of failing outright.

The rules live inside the index. Which workspace an item belongs to, whether it's allowed to be seen, whether it's gone stale, those checks happen inside the index itself. Off-limits or out-of-date items never enter the results in the first place.

Every record has one exact identity. Each record carries a single unambiguous identity, so an update or a delete lands on precisely the right record, and re-sending the same information changes nothing. Memory stays consistent even when the same thing arrives twice.

If memory breaks, the chat continues. When any part of this fails, it fails quietly. The assistant may lose some context, but the conversation keeps going. Memory should never be the reason a chat stops working.

What this means in practice

Most of this is invisible when it works, which is the point. In practice it means the assistant tends to have the context you actually gave it, the version of a fact it recalls is the current one. The relevant surrounding discussion comes back instead of a stray fragment, and something you deleted stays gone.

The principle we'd keep

The idea we took from this is easy to state and easy to violate: memory isn't a feature you add to a product. It's a system you design, and it has to be designed as one thing. Each time Libra's memory grew by accretion, one more feature storing one more thing its own way - it got less trustworthy, not more.

We rebuilt it around that principle. We expect to keep finding places where we got it wrong, and we'll write about those too.