{"data":{"kind":"file","path":"README.md","version_id":"aoqtic9338l744mznx1e0fx1","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2498,"modified_at":"2026-07-02T08:59:34.019000","content_hash":"89e57d4d27d44ba8bb54196276cbc8219f24972605f3ad6665c6e2bd54f6172e"},"entries":[],"content":"# glyph\n\nA verifiable-reward RL environment for a **Rust tool-use coding agent**. Each\ntask hands the model a real Rust crate and a job: patch until `cargo_test`\npasses, patch until `cargo_run` prints exact stdout, or confirm an\nalready-correct crate. The model emits `CALL tool {...}` blocks, tools execute\nagainst real `cargo`, and the episode must end with a clean `FINAL`.\n\nReward is verifiable — cargo actually compiles and runs, not an LLM judge.\nStrict success (`valid_trace`) requires terminal cargo success + one clean\n`FINAL` after it + exact `CALL` syntax + no tool use after success.\n\nFull write-up — diagnosis of a flat sparse-reward RLVR run and a dense\npartial-credit reward that measured a small (not statistically significant)\npass@8 lift over both SFT and the sparse control:\n<https://jayzenith.github.io/GLYPH/>. Source: <https://github.com/JayZenith/GLYPH>.\n\n## Requirements\n\nA Rust toolchain on `PATH` (`cargo`, `rustc`) — install via <https://rustup.rs>\nif `rustc --version` fails. Crate templates (~30MB) download automatically\nfrom the companion [`JayZenith/glyph-crates`](https://huggingface.co/datasets/JayZenith/glyph-crates)\ndataset on first use and are cached locally.\n\n## Usage\n\n```python\nimport verifiers as vf\n\nenv = vf.load_environment(\"glyph\")\n```\n\n```bash\nuv run vf-eval glyph -m <your-model> -n 20 -r 5\n```\n\nDense partial-credit reward shaping (see the write-up for why this matters —\na sparse reward silently discards the identically-failing part of the hard\ntail via zero-advantage filtering):\n\n```python\nenv = vf.load_environment(\n    \"glyph\",\n    progress_compile_bonus=0.5,\n    progress_test_frac_bonus=2.0,\n)\n```\n\n## Honest caveats\n\nThe held-out eval behind the write-up's numbers is 150 cases that cluster\ninto a handful of template families (config-merge precedence, enum-dispatch,\nleaderboard-ranking cover roughly half the set) — treat it as a smaller\neffective sample than n=150 suggests. The write-up's headline comparison is\npass@8 over three independent unseeded reruns per arm (the eval harness has no\nsampling-seed control), with a sparse-reward control arm that measured exactly\nflat vs SFT; the dense-reward lift (+3.7) is consistent in direction but not\nstatistically significant (prompt-level paired permutation p ≈ 0.14–0.16), and\neach reward arm is a single training run. Full methodology and raw per-rollout\neval data: [`JayZenith/Glyph-RLVR-Eval-Results`](https://huggingface.co/datasets/JayZenith/Glyph-RLVR-Eval-Results).\n","encoding":"utf-8","truncated":false,"total_bytes":2498},"status":null}