{"data":{"kind":"file","path":"README.md","version_id":"sq1wmeb19iiv9ojzi22ptu44","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":9488,"modified_at":"2026-11-18T07:47:58.196000","content_hash":"41f9802fc185df8d11a4ed6fcdc8df0e78c6a11c941438fb659090c3f4854191"},"entries":[],"content":"# convomem-arena\n\n**Conversational-memory tasks for agents, from [Salesforce ConvoMem](https://huggingface.co/datasets/Salesforce/ConvoMem), on verifiers v1.**\n\nMany memory tests hide one fact among a few lines of filler. convomem-arena hides it in **weeks of realistic conversation with the same person**, and asks whether an agent can find it, use only its latest version, respect what the user prefers, and admit when it was never said.\n\nEach task asks a question about a user's earlier conversations with an assistant: a fact the user stated, something the assistant recommended, a detail that later changed, a preference to respect, or something that was never said at all. The answer is buried in a history that also holds many unrelated conversations from the same person. ConvoMem has 75,336 such items across 100 personas; tasks are built on the fly from the pinned dataset, so a run downloads only the files it needs.\n\n## One task\n\nA real `changing_facts` task, abridged. Ten conversations of history, two of which carry evidence; the rest are ordinary chat from the same person:\n\n```json\n{\n  \"id\": \"changing_evidence/2_evidence/f8eb363d-…_Community_Manager/65/ctx10\",\n  \"category\": \"changing_facts\",\n  \"question\": \"What is the date for The National concert?\",\n  \"answer\": \"The concert has been postponed to April 10th of next year.\",\n  \"evidence_positions\": [5, 6],\n  \"evidence_messages\": [\n    \"…I finally managed to get tickets to see The National. They're playing here in SF on October 5th…\",\n    \"…the entire tour is being postponed due to a scheduling conflict. My new show date is April 10th next year.\"\n  ]\n}\n```\n\n\"October 5th\" was right once and is wrong now. The agent only ever sees the question and the history; the answer, evidence and positions stay hidden for scoring.\n\n## Three ways to deliver the history\n\nThe same tasks reach the agent in three ways, so long-context recall and agentic memory use are measured on identical data:\n\n| `memory_mode` | What the agent gets | Tests |\n|---|---|---|\n| `context` (default) | The full history in the prompt, using ConvoMem's official answer prompt | Long-context recall (the paper's baseline) |\n| `tools` | Only the question; the history sits behind a per-rollout MCP server with `memory_search` (BM25), `memory_read_conversation` and `memory_list_conversations` | Agentic retrieval: deciding what to look up, and when |\n| `files` | Only the question; the history is written to `memory/conversation_NNN.txt` | Memory as a filesystem, with the harness's own tools |\n\nFull documentation: **<https://rahmansunny071.github.io/convomem-arena/>** · Source: <https://github.com/rahmansunny071/convomem-arena>\n\n## Quick start\n\n```bash\nprime env install memcova/convomem-arena\n\n# check that tasks load (no model calls, no API key)\nvf-validate convomem-arena -n 6 --only-setup --runtime.type subprocess\n\n# run six episodes, one per category (agent and judge default to Prime Inference)\nvf-run convomem-arena -n 6 -r 1 -m <model> \\\n  --env.agent.harness.id null --env.agent.runtime.type subprocess\n```\n\nFully local, with one Ollama model as both agent and judge:\n\n```bash\nexport OLLAMA_FAKE_KEY=unused\nvf-run convomem-arena -n 6 -r 1 -m qwen3:30b-a3b \\\n  --client.base-url http://127.0.0.1:11434/v1 --client.api-key-var OLLAMA_FAKE_KEY \\\n  --env.agent.harness.id null --env.agent.runtime.type subprocess \\\n  --env.taskset.context-sizes 10 \\\n  --env.taskset.task.judge.base-url http://127.0.0.1:11434/v1 \\\n  --env.taskset.task.judge.api-key-var OLLAMA_FAKE_KEY \\\n  --env.taskset.task.judge.model qwen3:30b-a3b\n```\n\n## A first comparison\n\nThe same 12 tasks (2 per category, 30 conversations of history each), answered by local `qwen3:30b-a3b` and judged by the same model:\n\n| Setting | User facts | Assistant facts | Changing facts | Abstention | Preferences | Implicit | **Overall** |\n|---|---|---|---|---|---|---|---|\n| `context`, full history (52–57k tokens) | 1/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | **11/12** |\n| `tools` (prompts under 2.3k tokens) | 2/2 | 2/2 | 1/2 | 2/2 | 0/2 | 0/2 | **7/12** |\n| `context`, cut to 16k tokens by Ollama | 1/2 | 0/2 | 0/2 | 2/2 | 1/2 | 1/2 | **5/12** |\n\nThree things show up even at this size:\n\n- **When the history fits, reading it all wins.** With the full 52–57k tokens in context, the model answered 11 of 12.\n- **Tools find facts but miss what shapes an answer.** In `tools` mode the agent retrieved all the evidence for 11 of 12 questions and got every user and assistant fact right, but scored 0 of 4 on preferences and implicit connections: it searched for what was asked, not for the stated taste or circumstance that should have changed its recommendation.\n- **Abstention can't tell you memory works.** It scored 2/2 even with two thirds of the history silently cut, because \"I don't know\" stays right when the evidence is gone. Read it beside the fact categories, never alone.\n\nThis is 2 tasks per category and a model judging itself: indications, not a leaderboard. `vf-run` with a larger `-n` (a multiple of 6 weights every category equally, so the mean reward is a category-balanced score) and a stronger judge turns it into one.\n\n## Categories\n\n| Category | Tests | Graded against |\n|---|---|---|\n| `user_facts` | Recall of what the user said, possibly across several conversations | gold answer |\n| `assistant_facts` | Recall of what the assistant said or recommended | gold answer |\n| `changing_facts` | Only the latest value of something that changed | gold answer |\n| `abstention` | Recognising that the answer was never given | refusal is RIGHT |\n| `preferences` | Recommendations that respect stated preferences | rubric |\n| `implicit_connections` | Connecting indirect context | rubric |\n\nTasks are interleaved across categories, so `-n 6` already covers all six. Evidence is mixed into same-persona filler with ConvoMem's official algorithm, seeded per task, keeping evidence order (which defines the answer for `changing_facts`).\n\n## Scoring\n\n- **Reward `correct`** (weight 1.0): 1.0 when the category's ConvoMem judge says RIGHT. The judge prompts are copied verbatim from the reference implementation, one per category. A reply with neither RIGHT nor WRONG **raises** instead of scoring 0, so a broken judge surfaces as an error.\n- **Exact match, optional** (fact categories): `--env.taskset.task.fact-scoring exact_match` rewards `user_facts`, `assistant_facts` and `changing_facts` without a judge call. A reply matches when it contains the gold answer's names and numbers; replies that dump the history, or restate a superseded number on `changing_facts`, score 0. It agreed with the judge on all 36 judged replies checked. The other categories always use their judge, and the default stays `judge` for comparability with the paper. Either way it is recorded as the `exact_match` metric.\n- **Metrics** (`tools` mode, recorded, never rewarded): `memory_tool_calls`, `evidence_recall` (share of evidence conversations the agent retrieved) and `conversations_retrieved`. Low recall with a wrong answer is a search failure; high recall with a wrong answer is a reasoning failure.\n\n## Configuration\n\n| Flag | Default | Meaning |\n|---|---|---|\n| `--env.taskset.categories` | all six | Subset of categories |\n| `--env.taskset.evidence-counts` | all | Keep items with these evidence counts (1–6) |\n| `--env.taskset.context-sizes` | `1 10 30` | Conversations per history; one task per size per item (the paper uses up to 300) |\n| `--env.taskset.seed` | `0` | Seeds item order and every mix |\n| `--env.taskset.task.memory-mode` | `context` | `context`, `tools` or `files` |\n| `--env.taskset.task.max-turns` | `40` | Model calls per rollout |\n| `--env.taskset.task.fact-scoring` | `judge` | `judge` or `exact_match` for the three fact categories |\n| `--env.taskset.task.judge.model` / `.base-url` / `.api-key-var` | Prime Inference | The judge endpoint |\n\nThe full reference is in the [documentation](https://rahmansunny071.github.io/convomem-arena/reference/configuration/).\n\n## Notes\n\n- **Sandbox shell harnesses.** With `bash` or any shell-capable harness on the `subprocess` runtime, the agent's shell runs on the evaluator host, which also holds the Hugging Face cache with gold answers. Use the `null` harness for `context` and `tools`, and `docker` or `prime` for `files` mode.\n- **Keep the judge fixed** across compared runs; the rubric categories are especially sensitive to the judge model.\n- **Local models: check the context window.** Ollama silently cuts any prompt longer than its context setting, so a long history is truncated without an error. Thirty conversations came to 52–57k tokens in the comparison above; with a 16k window the model saw under a third of them, and its score fell from 11/12 to 5/12. Raise it (for example a Modelfile with `PARAMETER num_ctx 65536`) and confirm `prompt_tokens` in the trace matches the history size.\n- **Comparable in protocol, not item for item:** samples are drawn here rather than taken from ConvoMem's pre-mixed test cases.\n\n## Citation and license\n\nIf you use this environment, please cite ConvoMem: Salesforce AI Research, arXiv:2511.10523 (2025), BibTeX at <https://arxiv.org/abs/2511.10523>.\n\n- **Dataset:** [Salesforce/ConvoMem](https://huggingface.co/datasets/Salesforce/ConvoMem), **CC-BY-NC-4.0** (non-commercial use only).\n- **Judge prompts and mixing algorithm:** ported verbatim from [SalesforceAIResearch/ConvoMem](https://github.com/SalesforceAIResearch/ConvoMem), **Apache License 2.0**.\n","encoding":"utf-8","truncated":false,"total_bytes":9488},"status":null}