{"data":{"kind":"file","path":"README.md","version_id":"se1panz5fx2o0tysmvuewq1x","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3910,"modified_at":"2026-11-07T09:33:40.113000","content_hash":"a710b21cce8d57b9474ab77689591bdc4ce42fc5d39b02105fa676490104dd05"},"entries":[],"content":"# Backrooms: 3D clue-trail navigation\n\nA computer-use eval that frontier agents fail. The agent must cross a first-person 3D maze by reading real clues against decoys, room by room, and escape without one wrong doorway.\n\n![Backrooms](https://huggingface.co/datasets/jessicayjung75/backrooms-media/resolve/main/backrooms-sample.gif)\n\n**Play it in your browser:** https://backrooms.paprika3d.app/eval/\n\n### Overview\n- **Environment ID**: `jessicayjung/paprika3d-backrooms`\n- **Author**: Keeno Corp (made with paprika3d)\n- **Tags**: computer-use, multimodal, 3d, navigation, long-horizon, memory, eval, game\n- **Tasks here**: 4 public tasks: `clue-trail-8-public` (8 rooms), `public-trail-12` (12 rooms), `public-faint-16` (16 rooms, faint clues) and `public-timed-12` (12 rooms, timed). The 3 held-out trails run on unpublished seeds, scored on our host only (see below)\n- **Reward**: task score 1.0 when resolved (escaped through the exit with 0 wrong rooms and 0 debug uses). For RL, a dense, bounded shaping reward: +20 per true-path room, +1 per metre toward the next path doorway, up to +/-2 for facing it, +2 per metre toward the fire door in the exit room, +50 on escape, -5 per wrong doorway; turning or pacing nets zero\n\n### What it tests\n- **Long-horizon 3D navigation with memory:** 8 to 16 rooms, with a junction in every room. One wrong doorway fails the run, so the agent has to keep track of where it has been.\n- **Visual discrimination under decoys:** real clues (lit EXIT sign, clean evacuation map, fresh chalk arrow, worn carpet, ceiling lights lined up with a door) sit next to dead, faded, smudged or crossed-out fakes.\n- **Pixels in, state-graded out:** the agent sees only screenshots and acts through 7 computer tools. The grade reads the game's own state (escaped, wrong rooms = 0, debug uses = 0), so it can't be gamed with text output.\n- **Exactly reproducible:** a paused, seeded clock, so the same actions replay the same frames. Reference solution 1, no-op 0 on every task, and container privacy-checked (the agent can't read the level).\n\n### Results\n| Model (blind, computer tools only) | clue-trail-8-public | held-out 12-room | held-out 16-room faint | held-out 12-room timed |\n|---|---|---|---|---|\n| Claude Opus | 0/3 | 0/3 | 0/3 | 0/3 |\n| Claude Sonnet | 0/3 | 0/3 | 0/3 | 0/3 |\n| Claude Haiku | 0/3 | 0/3 | 0/3 | 0/3 |\n| Reference solution (scripted) | 1.0 | 1.0 | 1.0 | 1.0 |\n| No-op | 0.0 | 0.0 | 0.0 | 0.0 |\n\n### Quickstart\n```bash\nprime env install jessicayjung/paprika3d-backrooms\nvf-run paprika3d-backrooms -m <model> --env.agent.harness.id null --env.agent.runtime.type subprocess\n```\nThe evaluator needs `bun` and headless Chrome. Tools: `screenshot`, `click`, `drag`, `mouse`, `scroll`, `key`, `wait`.\n\n### Held-out set and full environment\n| Product | Price | How you get it |\n|---|---|---|\n| Public tasks (`clue-trail-8-public` and 3 public variants) | Free | HUD, Prime Intellect hub, Harbor; trajectories on Hugging Face |\n| Held-out set: 3 harder trails on unpublished seeds | $1,200 per task or $3,000 for all three (scoring licence) | Hosted: send a model endpoint, get scores back. The levels never leave us, so they stay uncontaminated |\n| Backrooms RL environment (seeded procedural maze + clue trail, `reset(seed)`/`step(action)`, HUD/Harbor/verifiers adapters, containerised) | $3,000 non-exclusive (exclusive on request) | Container + source package |\n\nQuestions and orders: **founders@keeno.dev**.\n\n### Licence and credits\nThis free sample is © 2026 Keeno Corp under **CC BY-NC 4.0** (`paprika3d_backrooms/LICENSE`): free for research and other non-commercial use with attribution. Commercial use, the held-out tasks, the full RL environment and paid data are under contract (founders@keeno.dev). Third-party models (Sketchfab, CC BY 4.0) and CC0 textures and sounds are credited in [`paprika3d_backrooms/CREDITS.md`](paprika3d_backrooms/CREDITS.md).\n","encoding":"utf-8","truncated":false,"total_bytes":3910},"status":null}