{"data":{"kind":"file","path":"README.md","version_id":"fucl11hk2vct4rk8w9pmvg8q","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3204,"modified_at":"2026-08-13T08:38:07.625000","content_hash":"9b46e70c717c1ab002721783424b2a31ca87c95d3e19c9970b918c7a603b886f"},"entries":[],"content":"# psychscanner-nback\n\n### Overview\n- **Environment ID**: `psychscanner-nback`\n- **Short description**: N-back working-memory task — judge whether the current letter matches the one shown `n` positions back, from [psychscanner-primal](https://github.com/saurabhr/psychscanner-primal)'s `nback_demo` task card. Replaces `psychscanner-rm-encoding` as the default registered task.\n- **Tags**: psychology, cognitive-science, working-memory, single-turn\n\n### Memory levels\nTwo independent axes, crossed in the bundled dataset:\n\n- **n-back level** (`n`): 1, 2, or 3 — how many letters back the model must compare against. Higher `n` = higher working-memory load.\n- **History-quantization condition** (`memory`), mirroring psychscanner's own `memory_k` / `summary_k` (see `docs/guides/memory_types.md` in the main [psychscanner](https://github.com/saurabhr/psychscanner) repo):\n  - `conversation` — the trailing `memory_k=5` letters are shown verbatim.\n  - `summary` — everything older than `summary_k=10` letters is folded into a per-letter count summary; the most recent `summary_k=10` letters are still shown verbatim.\n\n### Datasets\n- **Primary dataset**: `nback_demo.json`, bundled in this package (132 trials: 3 n-levels x 2 memory conditions x ~22 trials each, seeded/deterministic).\n\n### Task\n- **Type**: single-turn (each trial's prompt embeds the relevant slice of history per the memory condition above)\n- **Output format**: response wrapped in `<answer>match</answer>` / `<answer>no-match</answer>` tags (parsed with `verifiers.XMLParser`)\n- **Rubric overview**: one reward function, `nback_correct` — 1.0 if the match/no-match judgment equals the trial's ground truth, else 0.0.\n\n### Quickstart\nRun an evaluation with default settings (all n-levels, both memory conditions):\n\n```bash\nprime eval run psychscanner-nback\n```\n\nRestrict to one n-back level and/or memory condition via env args:\n\n```bash\nvf-eval psychscanner_nback --env-args '{\"n\": 2, \"memory\": \"summary\"}' \\\n  -m openai/gpt-4.1-mini -n 22 -r 3\n```\n\nLocally, against a self-hosted OpenAI-compatible endpoint (e.g. Ollama):\n\n```bash\nvf-eval psychscanner_nback --provider local \\\n  --api-base-url http://localhost:11434/v1 --api-key-var OLLAMA_API_KEY \\\n  -m smollm2:360m-instruct-fp16 -n 22 -r 1\n```\n\n### Metrics\n\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` / `nback_correct` | 1.0 if the match/no-match judgment is correct, else 0.0 |\n| `num_turns` | Always 1.0 — sanity check that this ran as single-turn |\n\n### Citation\n\nThis task is derived from psychscanner-primal, itself a slim distribution of [psychscanner](https://github.com/saurabhr/psychscanner). If you use this environment in research, cite:\n\n```bibtex\n@misc{ranjan2026reality,\n      title={Reality Monitoring in Large Language Models: Self-Knowledge That Transforms with Conversation Memory},\n      author={Saurabh Ranjan and Konstantina Sokratous and Brian Odegaard},\n      year={2026},\n      eprint={2607.23927},\n      archivePrefix={arXiv},\n      primaryClass={cs.AI},\n      url={https://arxiv.org/abs/2607.23927},\n}\n```\n\nFull citation list: [psychscanner-primal/CITATION.cff](https://github.com/saurabhr/psychscanner-primal/blob/main/CITATION.cff).\n","encoding":"utf-8","truncated":false,"total_bytes":3204},"status":null}