{"data":{"kind":"file","path":"README.md","version_id":"bqm56i1v68fdmw7bi6vccnfo","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5030,"modified_at":"2026-09-14T15:42:30.039000","content_hash":"1278faf32d43a104d1f6ff301cc1d7e31640281a8076289eac91ffba3d96b196"},"entries":[],"content":"# GlyphBench\n\n**A playground for language-model reinforcement learning.**\n\nGlyphBench brings popular games into a shared text interface for studying how\nlanguage models plan, remember, explore, and learn from interaction. Agents see\na two-dimensional Unicode grid, a legend, and a status display, then choose a\nnamed action. The same environments support evaluation, reinforcement learning,\nand interactive replay.\n\nThe standard benchmark contains **303 tasks across six suites**: Classics (50),\nMiniGrid (71), MiniHack (63), Craftax (60), Miniatari (43), and Procgen (16).\nStandard episodes finish in fewer than 512 environment turns and keep cumulative\nreturns in `[-1, 1]`. The games run on the CPU and do not require external services.\nLLM evaluation uses the model provider or inference server you supply.\n\nThere are also 59 extended tasks: Atari (57), CraftaxFull (1), and NetHack (1).\nThe full Craftax and NetHack games require optional native dependencies and\npreserve their native rewards. They are evaluated separately from the standard\nbenchmark.\n\n## Install\n\nUse Python 3.12 and uv. Install Prime as a separate tool, then create an environment\nfor GlyphBench:\n\n```bash\nuv tool install 'prime==0.6.35'\nmkdir glyphbench-work\ncd glyphbench-work\nuv venv --python 3.12\nsource .venv/bin/activate\nuv pip install --python .venv/bin/python \\\n  'verifiers @ git+https://github.com/PrimeIntellect-ai/verifiers.git@b2e4e81'\nprime env install roger-creus/glyphbench\n```\n\nKeep the Prime CLI in its tool environment. GlyphBench pins the Verifiers revision\nused by its native v1 adapter, while Prime 0.6.35 has a different Verifiers pin;\ninstalling both into the same Python environment causes a dependency conflict.\nThe bootstrap installs that required runtime first: Prime 0.6.35 only selects an\nactive project environment once it contains Verifiers. Without this step, its\ninstaller can target the CLI tool environment instead. Evaluation commands then\nuse the active GlyphBench environment.\n\n## Try a game\n\n```python\nfrom glyphbench.core import make_env\n\nenv = make_env(\"glyphbench/minigrid-empty-5x5-v0\")\ntry:\n    observation, info = env.reset(seed=42)\n    print(observation)\n    action = env.action_spec.index_of(\"MOVE_FORWARD\")\n    observation, reward, terminated, truncated, info = env.step(action)\n    print(observation)\nfinally:\n    env.close()\n```\n\nThe direct API accepts integer action indices. An LLM harness presents named\nactions and parses responses such as `<action>MOVE_FORWARD</action>`.\n\n```bash\nglyphbench list-suites\nglyphbench list-envs\n```\n\n## Evaluate with a model server\n\nWith the GlyphBench environment active, point Prime at your OpenAI-compatible\nmodel server. Replace `MODEL_NAME` and the URL, and set `OPENAI_API_KEY` to the\ncredential required by that server:\n\n```bash\nprime eval run glyphbench \\\n  --provider vllm -m MODEL_NAME \\\n  -b http://localhost:8000/v1 -k OPENAI_API_KEY \\\n  -n 1 -r 1 -c 1 -t 4096 \\\n  -a '{\"task_id\":\"glyphbench/classics-snake-easy-v0\",\"num_episodes\":1,\"max_turns\":16,\"use_memory\":false}' \\\n  --skip-upload --save-results --disable-env-server --disable-tui\n```\n\nThis is a short smoke evaluation. Remove `max_turns` to use the task's native\nhorizon. Select environments with either `task_id` or the `include_suites` and\n`include_tasks` filters; an explicit task ID cannot be combined with filters.\nFor a whole task selection, set `-n -1` to evaluate every generated task/seed row.\nResults remain local with `--skip-upload`.\n\nThe package also exports native Verifiers v1 environment, taskset, and harness\nclasses. From the `glyphbench-work` directory created above, `.venv/bin/eval`\nexercises those directly. This example uses the default Snake task:\n\n```bash\n.venv/bin/eval glyphbench \\\n  --model MODEL_NAME \\\n  --client.base-url http://localhost:8000/v1 \\\n  --client.api-key-var OPENAI_API_KEY \\\n  -n 1 --env.taskset.num-seeds 1 --env.player.max-turns 16 \\\n  --no-serve --no-push\n```\n\n## Source and training\n\nDownload the source, documentation, tests, and pinned training configurations:\n\n```bash\nprime env pull roger-creus/glyphbench --target glyphbench-source\ncd glyphbench-source\n```\n\nThe training recipes under `configs/rl/qwen35-4b-glyphbench/` use a coupled\nPrime-RL GPU stack. Follow that directory's guide and its frozen uv lock when\nreproducing training; installing optional GPU extras from the Hub wheel alone\ndoes not reproduce the repository's source and wheel-index settings.\nThe optional full-game backends need the external `third_party/Craftax` and\n`third_party/nle` checkouts at the revisions recorded in `pyproject.toml`.\nThe 303 standard games do not need those checkouts.\n`docs/DEPENDENCIES.md` describes the remaining upstream GPU advisories and their\nscoped mitigations. Local environment installation and rollouts are checked;\nPrime-hosted GPU training and evaluation have not been validated for this release.\n\nGlyphBench is distributed under the MIT license. The full-game adapters use their\nupstream projects under those projects' respective licenses.\n","encoding":"utf-8","truncated":false,"total_bytes":5030},"status":null}