{"data":{"kind":"file","path":"README.md","version_id":"ha94xrv1u5thlespdo72n1zh","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":4892,"modified_at":"2026-08-05T02:16:05.521000","content_hash":"55a28c8094ecb3f1a7f6266c37df88b36fcc7f9b1e3661227f89faf6a9407f63"},"entries":[],"content":"# router-frontier-rl\n\nTrains an RL router — given a SWE-bench task, pick which of Nemotron 3 Ultra /\nClaude Opus 4.7 / GPT-5.5 should attempt it — reproducing Applied Compute's\n[Training an Agentic Router](https://www.appliedcompute.com/research/training-an-agentic-router)\nreward exactly: **1.0** for the optimal (oracle) model choice, **0.0** for any\nother valid model, **-1.0** for no-route (unparseable output). No partial\ncredit for ties — the article found that collapses training toward\nalways-Opus.\n\nTrains **offline** against `oracle_table.json`, a precomputed table built by\n`../scripts/build_oracle_table.py` from this project's own real,\nharness-verified SWE-bench Verified `batch1` results (20 tasks). No live\nmodel or Docker-harness calls happen during training — same pattern as the\nReward Hacking Sprint's `cascade-router` env and its bundled\n`teacher_table.json`.\n\n## Oracle rule\n\nPer task: the best-performing model wins; ties broken by the **cheapest**\namong the tied models — applied uniformly, including when all three models\nfail (cheapest-of-the-failures is still the oracle pick). Matches the\narticle's construction exactly. See `oracle_table.json`'s `per_model` field\nper task for the real resolved/cost data behind each pick.\n\n## Deviations from the article (disclosed)\n\n1. **Scale**: 20 tasks, 1 rollout per model (real `batch1` data), not 497\n   tasks / 3 rollouts/model. Oracle \"performance\" per task is a single\n   resolved boolean per model, not a rate over 3 samples — ties are more\n   common here than in the article's setup.\n2. **GRPO hyperparameters**: Prime Intellect's Hosted Training config schema\n   (`prime train configs`) exposes `learning_rate`, `batch_size`,\n   `rollouts_per_example`, `max_steps`, and `[sampling].max_tokens`/\n   `temperature` — but not ε-clipping, ε_high, KL, entropy, Adam betas,\n   weight decay, or grad clip. Only `learning_rate=5e-7` from the article's\n   optimizer spec is settable; everything else is Prime's platform default.\n3. **Policy input**: a static system-prompt model-capability summary (from\n   our own frontier data — observed resolve rates, relative cost/turn-count\n   behavior) + the task's `problem_statement`/`repo`. No per-task dynamic\n   \"capability analysis\" LLM pre-pass, no bash-tool read-only repo access.\n   Single-turn only (`vf.SingleTurnEnv`).\n4. **No held-out split**: 20 tasks is too few to hold out a meaningful test\n   slice — trains on all of batch1, no in-training held-out eval.\n5. **Rollout shape**: `batch_size=20` (all of batch1 every step — with only\n   20 total tasks this is necessarily full-batch, not a 64-of-497 subsample\n   like the article) x `rollouts_per_example=8` (matches the article exactly)\n   = GBS 160, smaller than the article's 512 simply because there are fewer\n   tasks available — not rescaled up, to keep this cheap.\n6. **Base policy model**: the article doesn't specify one for the router\n   itself; `sprints/Llama-3.2-1B-Instruct` is used here — the free-queue\n   variant of `cascade-router`'s own established smoke-scale choice\n   (confirmed 2026-08-04: the bare `meta-llama/Llama-3.2-1B-Instruct` isn't\n   in Prime Hosted Training's available-models list at all).\n\n## Quick start\n\n```bash\nprime env install router_frontier_rl -p .    # from router-frontier-repro/ (note: local\n                                              # install needs the underscored folder name)\n\n# cheap eval-only sanity check (NOT training) — confirms the env loads,\n# prompts render, and routed_reward spans {-1, 0, 1} sanely (real 2026-08-04\n# run: 78/80 unparseable (-1.0), 2/80 matched oracle (1.0) with the few-shot\n# example below — an untrained model rarely uses the <route> tag unprompted)\nprime eval run router_frontier_rl -m meta-llama/Llama-3.2-1B-Instruct --provider prime \\\n  --num-examples 20 --rollouts-per-example 2\n\n# GRPO smoke test (3 steps) — confirms the Hosted Training loop itself\n# completes end-to-end against this env. Push first if the Hub copy is stale:\n# prime env push router_frontier_rl -p .. --visibility PRIVATE\nprime train sprint-config-smoke.toml --yes\n\n# the real run, only after the smoke test looks clean\nprime train sprint-config-batch1.toml --yes\n```\n\n## Environment arguments\n\n| Arg | Default | Meaning |\n|-----|---------|---------|\n| `table_path` | bundled `oracle_table.json` | Path to the oracle table JSON. |\n| `weights` | `[1.0, 0.0, 0.0, 0.0, 0.0]` | Reward weights: `[routed_reward, chose_oracle, parsed_valid_route, resolved_if_chosen, cost_if_chosen]` — only `routed_reward` is trained by default. |\n\n## Rebuilding the oracle table\n\nIf `batch1`'s underlying rollouts/eval results change, regenerate from\n`router-frontier-repro/`:\n\n```bash\npython scripts/build_oracle_table.py\n```\n\nThis overwrites `rl_env/oracle_table.json` from\n`build_results_table()`/`load_tasks()` — no manual editing of the table.\n","encoding":"utf-8","truncated":false,"total_bytes":4892},"status":null}