{"data":{"kind":"file","path":"README.md","version_id":"rg4woeaxqep6jsur9fmpjp46","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":4520,"modified_at":"2026-08-13T08:39:13.798000","content_hash":"61426d0b4f6735191d309723ea3a9265b29f275f2992c95370915c6df6759920"},"entries":[],"content":"# ps-primal\n\n### Overview\n- **Environment ID**: `ps-primal`\n- **Short description**: Paired-associate recall — score whether a model recalls the correct second word of a studied pair, across five word-pair semantic-similarity levels, from [psychscanner-primal](https://github.com/saurabhr/psychscanner-primal)'s `pal50` task card.\n- **Tags**: psychology, cognitive-science, memory, single-turn\n\n### Scope — please read before using results\nThis environment is a **single-turn proxy** for the real `pal50` paradigm, not a faithful replication:\n\n- **Covered**: for each of the 50 test pairs, does the model recall the correct `word2` given `word1`, when the pair is studied and tested in the same prompt? Results are tagged by `similarity` (0.0–1.0) so accuracy can be compared across interference levels.\n- **Not covered — the real design is interleaved and delayed.** In `pal50.json`, all 50 pairs are studied first (encoding phase), then all 50 are tested afterward (test phase), so recall happens after many *other* pairs have intervened — the actual manipulation is memory under interference/delay, not immediate recall. `verifiers` scores each dataset row as an independent single-turn rollout, so this environment collapses study+test into one prompt per item instead. That's a materially easier task than the source paradigm and will likely show higher, less separated-by-similarity accuracy than a real run.\n- A faithful port needs `MultiTurnEnv` (study all 50 pairs across turns, then test all 50, preserving conversation state throughout) — left for a follow-up, same as noted in\n  [`psychscanner-nback`](https://github.com/saurabhr/psychscanner-primal/tree/main/environments/psychscanner_nback)'s own README for this same task.\n\nIf you need the full paradigm, run `pal50.json` directly through [psychscanner-primal](https://github.com/saurabhr/psychscanner-primal)'s `ExpCard`/`ScannerModel` with `memory=\"Convo\"`, `chain_type=\"trial\"` rather than treating this environment's reward as a complete replication.\n\n### Datasets\n- **Primary dataset**: `pal50.json`, bundled in this package (100 items: 50 encoding + 50 matching test trials; only the 50 test trials become dataset rows, since each already carries its studied pair). Same file as `psychscanner-primal`'s `examples/tasks/pal50.json`.\n- **Source**: [psychscanner-primal/examples/tasks](https://github.com/saurabhr/psychscanner-primal/tree/main/examples/tasks)\n- **Split sizes**: 50 train, no separate eval split (dataset doubles as both) — 5 similarity levels x 10 pairs each.\n\n### Task\n- **Type**: single-turn\n- **Output format**: response wrapped in `<recall>...</recall>` tags (parsed with `verifiers.XMLParser`)\n- **Rubric overview**: one reward function, `recall_correct` — 1.0 if the recalled word exactly matches the studied `word2` (case-insensitive), else 0.0.\n\n### Quickstart\nRun an evaluation with default settings:\n\n```bash\nprime eval run ps-primal\n```\n\nConfigure model and sampling:\n\n```bash\nprime eval run ps-primal -m openai/gpt-4.1-mini -n 10 -r 3\n```\n\nLocally, against a self-hosted OpenAI-compatible endpoint (e.g. Ollama):\n\n```bash\nvf-eval ps_primal --provider local \\\n  --api-base-url http://localhost:11434/v1 --api-key-var OLLAMA_API_KEY \\\n  -m smollm2:360m-instruct-fp16 -n 10 -r 1\n```\n\n### Taskset Config\n\n| Field | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `similarity` (in `info`, not a config field) | float | — | 0.0/0.25/0.50/0.75/1.0 — the semantic-similarity level of each pair, for post-hoc slicing of accuracy by interference level |\n\n### Metrics\n\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` / `recall_correct` | 1.0 if the recalled word exactly matches the studied pair, else 0.0 (see Scope for why this is easier than the source paradigm) |\n| `num_turns` | Always 1.0 — sanity check that this ran as single-turn |\n\n### Citation\n\nThis task is derived from psychscanner-primal, itself a slim distribution of [psychscanner](https://github.com/saurabhr/psychscanner). If you use this environment in research, cite the framework paper:\n\n```bibtex\n@misc{ranjan2026psychscanner,\n      title={Psych Scanner: A Framework for Systematic Cognitive Evaluation of Large Language Models},\n      author={Saurabh Ranjan and Konstantina Sokratous and Mukesh Makwana},\n      year={2026},\n      note={Manuscript submitted for publication},\n}\n```\n\nFull citation list: [psychscanner-primal/CITATION.cff](https://github.com/saurabhr/psychscanner-primal/blob/main/CITATION.cff).\n","encoding":"utf-8","truncated":false,"total_bytes":4520},"status":null}