{"data":{"kind":"file","path":"README.md","version_id":"njcj4apn9qi79e3tjjwcqpbi","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2320,"modified_at":"2026-07-17T16:16:48.090000","content_hash":"acb7ed74f715407dd9ff0e50459a73f253bb11a34038ab45811a00a7bceae5ba"},"entries":[],"content":"# crayfish\n\n### Overview\n- **Environment ID**: `crayfish`\n- **Short description**: Train an LLM to play chess as White or Black against Stockfish, earning rewards for legal play, correct side, flagging opponent illegal moves, and winning.\n- **Tags**: `chess`, `rl`, `tool-use`, `stockfish`, `multi-turn`\n\n### Datasets\n- **Primary dataset(s)**: 7,496 opening FEN positions drawn from real games, covering a wide range of early-game scenarios.\n- **Source links**: N/A — curated from game databases.\n- **Split sizes**: 7,496 examples (all used for training/eval).\n\n### Task\n- **Type**: multi-turn tool use\n- **Output format expectations**: XML-tagged responses — `<reasoning>...</reasoning>` for chain-of-thought, then `<whitemove>e2e4</whitemove>` or `<blackmove>e7e5</blackmove>` with UCI long algebraic notation. Board state is communicated via `<boardstate>...</boardstate>` FEN strings.\n- **Rubric overview**:\n  - `plays_correct_side` (0.5) — penalizes playing the opponent's color\n  - `correctly_flags_opponent_illegal_moves` (0.5) — rewards detecting when the opponent plays illegally\n  - `plays_legal_and_valid_move` (0.5) — penalizes illegal moves from the assistant\n  - `wins_game` (1.0) — rewards checkmate, penalizes losing or drawing\n\n### Quickstart\nRun an evaluation with default settings:\n\n```bash\nprime eval run crayfish\n```\n\nConfigure model and sampling:\n\n```bash\nprime eval run crayfish   -m openai/gpt-4.1-mini   -n 20 -r 3 -t 1024 -T 0.7   -a '{\"key\": \"value\"}'  # env-specific args as JSON\n```\n\nNotes:\n- Use `-a` / `--env-args` to pass environment-specific configuration as a JSON object.\n\n### Environment Arguments\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `stockfish_path` | str | `/usr/local/bin/stockfish` | Path to the Stockfish engine binary |\n| `stockfish_time` | float | `0.1` | Time limit (seconds) per Stockfish move |\n\n### Metrics\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` | Weighted sum of all reward criteria |\n| `plays_correct_side` | Whether the assistant only moved pieces of their assigned color |\n| `correctly_flags_opponent_illegal_moves` | Whether illegal opponent moves were caught |\n| `plays_legal_and_valid_move` | Whether the assistant avoided illegal moves |\n| `wins_game` | Whether the assistant won by checkmate |\n","encoding":"utf-8","truncated":false,"total_bytes":2320},"status":null}