{"data":{"kind":"file","path":"README.md","version_id":"gh19m2uznzltvray0sewde6h","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":8213,"modified_at":"2026-07-06T02:03:02.695000","content_hash":"53d5cc8d280b1e67d26c9d0814649ab418f6b2f4e6466d5be17b7001373c87d8"},"entries":[],"content":"# gyrell (env)\n\n### Overview\n- **Environment ID**: `gyrell` (hub: `lumpenspace/gyrell`)\n- **Short description**: Codenames-style hidden-information word game. The policy plays the red team — cluegiver and guesser seats, each with its own private conversation — against a scripted opponent team.\n- **Tags**: game, multi-turn, hidden-information, train, eval\n\n### Datasets\n- **Primary dataset(s)**: procedurally generated boards from the `turngames` Codewords engine (`classic-en` deck), seeded and fully deterministic. Each example is one game: a distinct board, key, and starting team (alternating first/second move).\n- **Source links**: [lumpenspace/codenames-llms](https://github.com/lumpenspace/codenames-llms) — the same engine drives the live broadcast arena, replays, and this environment.\n- **Split sizes**: configurable; defaults to 2000 train / 200 eval.\n\n### Task\n- **Type**: multi-turn (non-linear: per-seat private message histories)\n- **Output format expectations**: one JSON object per turn, no fences:\n  `{\"public_deliberation\": \"...\", \"action\": {\"type\": \"give_clue\", \"word\": \"ocean\", \"count\": 2}}`\n- **Rubric overview**: trajectory-level rewards from the final game outcome — earned win (1.0), earned own-word progress (0.3), penalty-only legality (0.2) and format (0.1) terms — plus zero-weight metrics (raw wins, assassin losses, game completion, turns played). See Metrics below for the rationale.\n\n### How it works\nThe trained policy always plays the **red** team. With `role=\"team\"` (default) it\noccupies both the cluegiver and guesser seats; because the cluegiver sees the\nhidden key and the guesser must not, **each seat keeps a private conversation**\n(implemented via the `get_prompt_messages` override — trajectory steps\ninterleave seats, and the guesser's context never contains the key).\n\nEvery seat the policy does not control is played by a configurable driver:\n\n- `\"scripted\"` (default): the deterministic random-but-legal baseline from\n  `turngames` — cheap and stationary, good for early training;\n- `\"policy\"`: the same model/client being trained (frozen for those turns —\n  they are environment transitions, not trajectory steps), i.e. self-play;\n- an **endpoint spec**: any OpenAI-compatible endpoint, e.g.\n  `{\"model\": \"openai/gpt-4.1-mini\", \"base_url\": \"https://openrouter.ai/api/v1\",\n  \"api_key_var\": \"OPENROUTER_API_KEY\", \"temperature\": 0.7, \"max_tokens\": 400}`.\n\n`opponent` sets the driver for the blue team; `partner` sets it for red seats\nthe policy does not control (relevant for `role=\"cluegiver\"`/`\"guesser\"` —\ne.g. train the cluegiver with a fixed strong guesser). Both also accept:\n\n- a **role-split dict** assigning different models per role —\n  `{\"cluegiver\": {\"model\": \"anthropic/claude-haiku-4.5\"}, \"guesser\": {\"model\": \"openai/gpt-4.1-mini\"}}`;\n- a **pool** (list) of drivers/role-splits — one matchup is chosen per\n  example, deterministically from the board seed, so all GRPO rollouts of an\n  example face the same opponents. The chosen matchup is recorded in\n  `game_result[\"seat_models\"]` for per-matchup analysis.\n\nLLM-driven seats keep their own private conversations (a blue guesser never\nsees any key), and any unusable reply or endpoint failure falls back to the\nscripted baseline for that turn (counted in the `other_seat_fallbacks`\nmetric), so a misbehaving opponent can never stall a rollout.\n\nIllegal actions (e.g. a clue matching a board word) are not silently repaired:\nthe canonical arbiter rules on them, applies the in-game penalty (an illegal\nclue ends the turn), and the ruling is reported back to the policy — so\nlegality is learned, not enforced by the harness. Unparseable replies get a\nformat reminder and count against `format_compliance`.\n\n### Quickstart\n\nRun an evaluation with default settings:\n\n```bash\nuv run vf-eval gyrell\n```\n\nConfigure model and sampling (OpenRouter example, as used in this repo):\n\n```bash\nuv run vf-eval gyrell \\\n  --provider openrouter -m \"openai/gpt-4.1-mini\" \\\n  -n 5 -r 2 --max-tokens 400 -T 0.7 \\\n  -a '{\"role\": \"team\", \"num_eval_examples\": 20, \"max_turns\": 40}'\n```\n\nPlay against a real LLM opponent instead of the scripted baseline:\n\n```bash\nuv run vf-eval gyrell \\\n  --provider openrouter -m \"openai/gpt-4.1-mini\" \\\n  -n 5 -r 2 --max-tokens 400 -T 0.7 \\\n  -a '{\"opponent\": {\"model\": \"anthropic/claude-haiku-4.5\", \"base_url\": \"https://openrouter.ai/api/v1\", \"api_key_var\": \"OPENROUTER_API_KEY\"}}'\n```\n\n### Environment Arguments\n\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `role` | str | `\"team\"` | Seats the policy controls: `team` (cluegiver + guessers), `cluegiver`, or `guesser`. |\n| `opponent` | str \\| dict \\| list | `\"scripted\"` | Driver(s) for the blue team: `\"scripted\"`, `\"policy\"` (self-play with the trained model's own client), an endpoint spec, a role-split dict, or a pool (list) of these — one matchup per example (see above). |\n| `partner` | str \\| dict \\| list | `\"scripted\"` | Driver(s) for red seats the policy does not control (only used with `role=\"cluegiver\"`/`\"guesser\"`); same forms as `opponent`. |\n| `guessers_per_team` | int | `1` | `1` uses the simple choose/stop action space; `2+` enables the propose/confirm protocol between guesser seats. |\n| `deck` | str | `\"classic-en\"` | Word deck from `turngames`. |\n| `num_train_examples` | int | `2000` | Number of seeded boards in the train split. |\n| `num_eval_examples` | int | `200` | Number of seeded boards in the eval split. |\n| `seed` | int | `0` | Base seed for boards and scripted opponents. |\n| `max_turns` | int | `50` | Cap on policy turns per rollout (a full game usually takes 20–40). |\n| max_consecutive_says | int | 2 | Cap on consecutive non-passing/deliberating actions (e.g. \"say\") to prevent stalling. |\n| `endpoint_seat_memory` | str | `\"digest\"` | How endpoint seats see the game: `\"digest\"` — stateless per call, a compact clue/reveal record rebuilt from the event log (the guesser role is nearly Markovian; ~1/20th the tokens, and cache keys become path-independent — measured hit rate 44% vs 24%); `\"conversation\"` — full growing per-seat history, for stateful partners (e.g. an endpoint cluegiver that should remember its intent). |\n| `cache_endpoint_seats` | bool | `true` | Memoize endpoint-seat responses on (endpoint, exact seat history) with in-flight dedup. Seat conversations are deterministic in the transcript prefix, so rollouts of the same board share adversary turns until the policy diverges — GRPO groups pay for one opening, not eight. Never applied to `\"policy\"` seats (their weights change every step). Cache hits are counted in `game_result[\"endpoint_cache_hits\"]`. |\n\n### Metrics\n\nTwo rules, each learned from a real reward hack observed in training:\n**only outcomes the policy can influence pay** (rewarding raw wins taught a\npolicy to forfeit every turn and let a weak opponent self-destruct — 100%\nwins, zero legal moves), and **protocol quality is a penalty, never income**\n(flat compliance bonuses taught a self-play policy to filibuster with legal\n`say` actions forever). A consecutive-`say` quota (`max_consecutive_says`)\nadditionally caps stalling structurally.\n\n| Metric | Weight | Meaning |\n| ------ | ------ | ------- |\n| `reward` | — | Weighted sum of the criteria below |\n| `win_earned` | 1.0 | 1 if red won by revealing its own last word |\n| `earned_progress` | 0.3 | Fraction of red's words revealed by red's own play |\n| `illegal_action_penalty` | 0.2 | 0 if all actions legal, down to −1 |\n| `format_penalty` | 0.1 | 0 if all turns parsed, down to −1 |\n| `action_legality` | 0.0 | Share of submitted actions ruled legal by the arbiter |\n| `format_compliance` | 0.0 | Share of turns with a parseable JSON action |\n| `team_win` | 0.0 | 1 if the red team won (any reason, incl. opponent mistakes) |\n| `word_progress` | 0.0 | Fraction of red's words revealed by anyone |\n| `assassin_loss` | 0.0 | 1 if red lost by revealing the assassin |\n| `game_finished` | 0.0 | 1 if the game reached a natural end within `max_turns` |\n| `turns_played` | 0.0 | Game turns (team alternations) played |\n| `other_seat_fallbacks` | 0.0 | LLM-driven non-policy turns that fell back to the scripted baseline |\n","encoding":"utf-8","truncated":false,"total_bytes":8213},"status":null}