{"data":{"kind":"file","path":"README.md","version_id":"ks62ih3g6for7nla2sc4pvr0","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5725,"modified_at":"2026-08-26T11:57:09.589000","content_hash":"f46d40d7715d1385e6c0deb83a9fe00fb2138efac6bee40a309e88c3244a5cc8"},"entries":[],"content":"# MemoryArena\n\nA [Prime Intellect](https://docs.primeintellect.ai/verifiers/v0/overview) environment for evaluating **memory-capable agents**.\n\nThe agent is not tested on knowledge. It is tested on whether it can\n**maintain a memory state**:\n\n```text\nRemember -> Update -> Forget -> Retrieve -> Reason over memory\n```\n\nMost memory benchmarks ask *\"can you remember?\"*. MemoryArena asks\n*\"can you maintain a memory state?\"* — facts change, corrections arrive,\ntimelines move, and only the newest truth counts.\n\n---\n\n## Task Categories (500 tasks)\n\n| Category          | Tasks | What the agent must do                                        |\n| ----------------- | ----- | ------------------------------------------------------------- |\n| `recall`          | 100   | Hold a fact across 5–100 noise turns                          |\n| `update`          | 100   | Latest fact wins (\"I moved to Bangalore\" beats \"Chennai\")     |\n| `contradiction`   | 100   | Apply explicit corrections (\"actually, it's June 15\")         |\n| `temporal`        | 100   | Track values across days; answer \"current\" or \"on Tuesday\"    |\n| `relational`      | 100   | Derive who manages whom from an out-of-order org chart        |\n\nEvery rollout follows the Prime Environment style:\n\n```text\nConversation starts -> facts introduced -> noise inserted -> facts updated /\ncorrected -> more noise -> question asked -> agent answers -> verifier scores\n```\n\n## Layout\n\n```text\nmemory-arena/\n├── pyproject.toml         # packaging for the Environments Hub\n├── generator.py           # regenerate all datasets deterministically\n├── memory_arena/          # installable package (importable as `memory_arena`)\n│   ├── __init__.py        # exposes load_environment()\n│   ├── environment.py     # verifiers env entrypoint\n│   ├── dataset/\n│   │   ├── recall.jsonl       # 100 tasks per file, seeded & reproducible\n│   │   ├── update.jsonl\n│   │   ├── contradiction.jsonl\n│   │   ├── temporal.jsonl\n│   │   └── relational.jsonl\n│   ├── tasks/             # task builders (one module per category)\n│   └── verifier/\n│       ├── scoring.py     # normalization + exact/containment matching\n│       └── metrics.py     # per-category metrics + weighted Memory Score\n└── examples/\n    ├── run_offline.py     # dependency-free pipeline sanity check\n    └── sample_tasks.py    # pretty-print one task per category\n```\n\nEach dataset row is one task:\n\n```json\n{\n  \"id\": \"recall-0042\",\n  \"category\": \"recall\",\n  \"noise_turns\": 40,\n  \"turns\": [{\"role\": \"user\", \"content\": \"...\"}, ...],\n  \"question\": \"What is my dog's name?\",\n  \"answer\": \"Luna\",\n  \"aliases\": [],\n  \"meta\": {\"slot\": \"pet_name\"}\n}\n```\n\n## Quickstart\n\n### Offline sanity check (no dependencies, no API keys)\n\n```bash\npython3 examples/run_offline.py\n```\n\nRuns two baseline agents over all 500 tasks and prints metric reports.\nThe `stale-fact` baseline demonstrates the point of the benchmark: it aces\nstatic recall and relational reasoning but scores **0.0 on update and\ncontradiction**, because it never applies newer information.\n\n### Evaluate with verifiers\n\n```bash\npip install 'verifiers>=0.1.8'\n\nvf-eval memory-arena -m gpt-4.1-mini -n 20\n# or programmatically:\npython -c \"\nimport verifiers as vf\nfrom memory_arena import load_environment\nenv = load_environment()\n\"\n```\n\n### Regenerate datasets\n\nDeterministic — same seed, same tasks:\n\n```bash\npython3 generator.py --n 100 --seed 42 --out dataset\n```\n\n## Memory Tool Mode\n\nThe interesting comparison is not model vs. model — it's **memory\narchitecture vs. memory architecture**:\n\n```python\nenv = load_environment(memory_tool=True)\n```\n\nIn this mode the agent additionally gets persistent per-rollout tools:\n\n| Tool              | Effect                              |\n| ----------------- | ----------------------------------- |\n| `store_memory`    | Write/overwrite a key-value fact    |\n| `retrieve_memory` | Read a stored fact                  |\n| `list_memories`   | Dump everything currently stored    |\n\nRun both modes for every agent and compare:\n\n| Agent            | No Memory Tool | Memory Tool |\n| ---------------- | -------------- | ----------- |\n| GPT              |                |             |\n| Claude           |                |             |\n| Qwen             |                |             |\n\nNow you're evaluating memory architectures, not just models.\n\n## Metrics\n\nNot just accuracy:\n\n| Metric                | Meaning                                    |\n| --------------------- | ------------------------------------------ |\n| Recall@10 / @50 / @100 | Accuracy after 10 / 50 / 100 noise turns  |\n| Update Accuracy       | Handles changed facts (latest-wins)        |\n| Contradiction Score   | Uses corrected truth                       |\n| Temporal Score        | Tracks day-by-day timelines                |\n| Relational Score      | Memory + multi-hop reasoning               |\n| **Memory Score**      | Weighted average: recall .20, update .25, contradiction .20, temporal .15, relational .20 |\n\nScoring (`verifier/scoring.py`) is deterministic: normalize both sides,\naccept an exact match or a short containment match, everything else is a miss.\nNo LLM judge in the loop.\n\n## Publishing to the Environments Hub\n\n```bash\nprime login\ncd ~/Desktop/rl/memory-arena\nprime env push\n```\n\n## Roadmap\n\n- Multi-turn delivery: drip-feed context turn by turn instead of one prompt\n- Forgetting pressure: distractor facts that later become relevant\n- Longer horizons: Recall@200+ with hierarchical summarization pressure\n- More agents on the leaderboard (OpenHands, LangGraph, mem0-backed)\n","encoding":"utf-8","truncated":false,"total_bytes":5725},"status":null}