{"data":{"kind":"file","path":"README.md","version_id":"sri31jqwgbeffex5twgt4vxf","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2455,"modified_at":"2026-10-13T14:46:01.577000","content_hash":"f9ed9db33fafa31a9aab46e14afa4f031f5b627b3c1597112f8226c0e11141ae"},"entries":[],"content":"# Agent Supermemory Analysis — V1.2.0\n\nAn iterative Verifiers benchmark for evidence-grounded reasoning over an agent's synthetic long-term memory audit trail. The default episode has two turns: a rehearsal memory question, scored feedback including its ground truth, then a distinct transfer question. Only the transfer answer affects reward. No external memory store or retrieval service is involved.\n\n## What it tests\n\nEight balanced case families: explicit corrections, as-of queries preceding a future correction, retractions, absence of a confirmed decision, alias resolution requiring two citations, re-confirmation after retraction, cross-field decoys, and retractions after the requested date. Every case also includes cross-project information, unconfirmed assistant suggestions, imported summaries, shuffled event order, and a prompt-injection decoy. Project names, fields, values, dates, and seeds vary deterministically.\n\nThe agent must answer with exact JSON, for example `{\"value\":\"jade\",\"evidence\":[\"E10000-6\"]}`. A retraction is cited when explaining that no current preference exists. When no decision has ever existed, use `{\"value\":null,\"evidence\":[]}`. The reward is 1.0 for a correct value with minimal evidence, 0.25 for a correct value with wrong evidence, and 0.0 for an incorrect value, malformed output, or duplicate citations. The grader checks only the final turn; the rehearsal ground truth does not reveal the transfer answer.\n\nPublic train and eval splits use disjoint seed ranges but the same eight case families. Thus the eval split measures within-template transfer, not hidden-task generalization. This is a synthetic benchmark, not persistent memory, live retrieval, real user data, or real-model performance.\n\n## Run locally\n\n```bash\nuv venv --python python3.12 .venv\nuv pip install --python .venv/bin/python -e . pytest\n.venv/bin/python -m pytest -q tests\nuv build --wheel\n```\n\n`load_environment(split=\"train\", size=64)` and `load_environment(split=\"eval\", size=32)` configure dataset size (1..256). Each dataset row includes `prompt`, transfer `answer`, and per-episode `info`. Unit tests cover 320 generated cases with an independently parsed chronological oracle and two real Verifiers rollouts driven by a scripted client (not by a model). The environment does not claim transfer to unseen task structures; adding independent human-curated transcripts and real-model baselines would be the next research step.\n","encoding":"utf-8","truncated":false,"total_bytes":2455},"status":null}