Aethra Moonshot Hackathon 2026Zero-to-one · AI/Compute

Mnemosyne

A sparse memory substrate that lets language models learn continuously — without retraining.

Transformer attention is a local-context operator being misused as a global-memory operator. Mnemosyne splits the two primitives: sparse content-addressable retrieval with an async consolidation loop, mimicking the hippocampal-neocortical division that biological brains evolved half a billion years ago.

256-dim
Sparse key
6.25%
Active dims per key
O(n·k)
Retrieval complexity
∞
Memory buffer size
The problem

Attention is the wrong primitive for memory.

Dense self-attention scales O(n²) with context length, which is why context windows are measured in thousands of tokens rather than millions of pages. Attention is also stateless — every inference forgets everything outside the window. And it cannot distinguish what was just perceived from what was consolidated long ago.

The industry response has been to scale: longer context, larger models, more RAG. This treats the symptom, not the disease. The disease is that attention is being asked to do a job it was never designed for.

The insight

Split the two primitives.

Biological brains solved this half a billion years ago. The hippocampus uses sparse, content-addressable episodic memory for recent experience and a separate consolidation pass to migrate episodic traces into semantic neocortical memory during sleep.

Mnemosyne makes the same split: attention stays local and dense, memory is sparse and content-addressable with async consolidation. The two communicate through a learned gate. The "context window" is no longer the unit of memory — the buffer is.

Thesis

"Transformer attention is a local-context operator being misused as a global-memory operator; Mnemosyne splits the two primitives and demonstrates that lifelong-learning language models are computationally tractable without retraining."

Three-stage data flow

Architecture

Three differentiable stages: encode (sparse projection + episodic write), retrieve (MIPS over key-value pairs + async consolidation pass + gated read), generate (context fusion + frozen LM head + lifelong update). The decoder is frozen — only the sparse projection and retrieval gate are trained.

1
Encode
Token Stream
Raw context. Frozen embedding via pretrained encoder.
Sparse Projection
Learned top-k mask. Outputs content-addressable key + value. Sparsity target: 6.25% active dims.
Episodic Write
Append to memory buffer. O(1) amortized. No gradient through buffer.
2
Retrieve
Sparse Memory Index
Maximum-inner-product search over key-value pairs. CA3 analog. Replaces O(n²) attention with O(n·k), k≪n.
Consolidation Pass
Async k-means clusters episodic entries into semantic prototypes. Mimics sleep consolidation.
Read & Gate
Soft top-k retrieval. Learnable gate decides memory vs context. Trained end-to-end.
3
Generate
Context Fusion
Concatenate retrieved memory values with local context. Single linear projection.
Frozen LM Head
Pretrained decoder generates next token. We do NOT fine-tune the decoder.
Lifelong Update
Every inference writes to memory. The system learns continuously without retraining. The zero-to-one claim.
Live prototype

Try Mnemosyne in your browser

Write memories, query them, and watch the consolidation pass collapse episodic traces into semantic prototypes. The sparse key visualization shows the 6.25% active dims per entry.

Write to Memory
Encode text into a sparse key and append it to the episodic buffer.
0 chars · produces 256-dim key with 16 active dims (6.25% sparsity)
Query Memory
Sparse MIPS retrieval with learned gating. Trusted memories are fused into context.
Top-5 retrieval · gate threshold 0.35
Consolidation Pass
Async k-means over the buffer. Mimics hippocampal sleep consolidation — episodic traces collapse into semantic prototypes.
Retrieved Memories
Top-k by cosine similarity · green badge = passed the gate (trusted for fusion)
No queries yet. Write some memories, then run a query to see sparse retrieval in action.
Memory Buffer
All episodic + semantic entries. Click any entry to inspect its sparse key.
Buffer is empty. Load seed memories or write your own.
Empirical evidence

Benchmark: sparse vs dense retrieval

Head-to-head against a vanilla bag-of-words attention baseline. Both systems query the same buffer of 8 seed memories across 8 long-context QA questions. The benchmark runs live in your browser — click "Run benchmark" to see it.

Single-Exposure Learning DemoHeadline result
Mnemosyne learns a new fact from a single write — no retraining. Vanilla attention cannot do this; it would need to be retrained on the new data. This is the zero-to-one claim.
Quantitative Benchmark: Mnemosyne vs Vanilla Attention Baseline
10 long-context QA questions. Score = 1.0 if the relevant keyword appears in the top-3 retrieved memories. Both systems query the same buffer.
Required deliverable

The Moonshot Paper

The paper is the central intellectual artifact. Below is its outline — ~4,200 words across 8 sections plus references. The full paper PDF is included in the submission package.

SectionWordsKey arguments
Abstract200Thesis statement. Problem (attention is wrong memory primitive), insight (split the two), architecture (sparse retrieval + consolidation), implication (civilization-scale shift).
1. Problem Humanity Misunderstood500Attention is misused as memory. Three structural limitations. Biological brains solved this 500M years ago.
2. Why Existing Solutions Are Insufficient400RAG, sparse attention, memory-augmented networks, long-context transformers — each addresses one symptom, none addresses the disease.
3. First-Principles Insight500Memory and attention are different primitives. Memory is sparse, content-addressable, append-only, with async consolidation.
4. Scientific & Technical Foundations800Kanerva 1988, Rolls & Treves 1998, Vaswani 2017, Lewis 2020, Rae 2016, Graves 2014.
5. Architecture & Method600Three stages: encode, retrieve, generate. Only projection and gate trained; decoder frozen.
6. Preliminary Results400Comparison against vanilla attention on long-context QA. Even modest wins are reportable because the claim is the principle, not the benchmark.
7. Long-Term Implications500Personal AI. 10–100× cost reduction. Alignment partially dissolves via preference stability.
8. The Future We Attempt to Create300The future is not bigger transformers. The future is models that remember.
References—APA. 8+ cited works.