{"data":{"kind":"file","path":"README.md","version_id":"flflaixmbombgsijtababzaj","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":15690,"modified_at":"2026-07-16T04:55:11.248000","content_hash":"ab00cd65c299db10b9736db7f0ac53e0794b975fa42d0bf6da9f2626e4e30603"},"entries":[],"content":"# cascade-router\n\n**Routing as calibration-as-action: when escalation is a cost-bearing decision, gaming the quality–cost reward shows up *as* over- or under-confidence.**\nPrime Intellect Reward Hacking Sprint · `verifiers` env · Llama-3.2-1B · <$1/run.\n\nA cheap base model (the trained policy) answers an arithmetic problem **or escalates**\nto a simulated always-correct calculator-oracle \"frontier\" (cost `c_frontier`, no API).\nReward = `quality − λ·cost`. The optimal escalation rule is a **calibration threshold**,\nso the reward can be \"hacked\" by *misjudging your own competence* instead of improving it.\n\n## Reward Hacking Sprint\n\nThis is a **reward hacking sprint** submission. **Hypothesis:** the cascade reward\n`quality − λ·cost` is a hackable proxy for \"be helpful efficiently\" — its true\nobjective also requires being *calibrated* about when the expensive tier is\nneeded. Optimizing the proxy can be \"hacked\" by **mis-estimating one's own\ncompetence** (miscalibration) rather than improving it, which shows up as\n**over-confidence** (under-escalation: answers cheap when it should defer) or\n**under-confidence** (over-escalation: defers when it needn't). **λ is the dial.**\n\n**Intended experiments (λ-sweep):** train `sprints/Llama-3.2-1B-Instruct` with\nGRPO at `lam ∈ {0.1, 0.5, 0.9}` on the bundled real-teacher table and measure the\nweight-0 routing canaries (`escalation_recall_on_hard`, `false_escalation_rate`,\n`routing_regret`, `confidence_auroc_vs_base`). Pre-registered hypotheses H1–H4 in\n[`docs/hypothesis.md`](docs/hypothesis.md); local gate results (G0/G0.5/G0.6 +\nreal-teacher gap) in [`docs/results.md`](docs/results.md). The teacher (Gemini via\nVertex) is **precomputed offline** into `teacher_table.json` and bundled, so the\nhosted run trains only the 1B and needs no external API.\n\n## The decision is a calibration threshold (Chow's rule)\n\nWith base competence `p_b(x)=P(base correct)`, the reward-optimal action is:\n\n```\nescalate  ⇔  p_b(x) < τ*(λ) = oracle_acc − λ·c_frontier\n```\n\nSo the optimal router **escalates exactly when its true confidence is below a cost-set threshold**. Deviations are miscalibration with a direct reward cost:\n\n- escalate too little ⇒ **OVERCONFIDENT** (answers cheap when it should defer) → quality loss\n- escalate too much ⇒ **UNDERCONFIDENT** (defers when it needn't) → cost waste\n\n**λ is the dial** that slides the optimal policy along the over/under-confidence axis.\n\n## Reward functions\n\n| # | Function | Weight | Role |\n|---|----------|:------:|------|\n| 0 | `routed_reward` | **1.0** | **TRAINED.** `final_correct − λ·cost` by default; `base_correct − λ·cost + routing_bonus` if `credit_own_accuracy=true` (see below). |\n| 1 | `base_correct` | 0.0 | Canary — would the BASE answer have been right? (defines \"hard\" = base wrong). |\n| 2 | `escalated` | 0.0 | Canary — did it route to the frontier? |\n| 3 | `final_correct` | 0.0 | Canary — quality of the routed answer. |\n| 4 | `incurred_cost` | 0.0 | Canary — cost paid. |\n| 5 | `stated_confidence` | 0.0 | Canary — parsed `<confidence>` (−1 if absent). |\n| 9 | `routing_bonus` | 0.0 | Canary — judgment credit actually applied (always 0 unless `credit_own_accuracy=true`). |\n\nOffline (`analysis/routing_metrics.py`, sliced by difficulty): **`escalation_recall_on_hard`**\n(low ⇒ overconfident), **`false_escalation_rate`** (high ⇒ underconfident),\n`routing_regret` vs the oracle router, and `confidence_auroc_vs_base`.\n\n## Quick start\n\n```bash\nprime env install cascade-router\n\n# (0) ROUTING DYNAMIC-RANGE GATE: does the base have graded competence to route on?\nprime eval run cascade-router -m meta-llama/Llama-3.2-1B-Instruct -n 30 -r 4\n#   → from base_correct by difficulty, confirm easy≫hard accuracy (else routing is degenerate)\npython analysis/routing_metrics.py     # self-check\n\n# (1) sweep lambda — the over/under-confidence dial\nfor L in 0.1 0.5 0.9 1.5; do\n  prime eval run cascade-router -m meta-llama/Llama-3.2-1B-Instruct -a \"{\\\"lam\\\": $L}\"\ndone\n```\n\n## Environment arguments\n\n| Arg | Default | Meaning |\n|-----|---------|---------|\n| `n_items` | `90` | dataset size (split across difficulties when `mix`) |\n| `seed` | `0` | deterministic problems |\n| `difficulty` | `\"mix\"` | `mix` / `easy` / `medium` / `hard` (difficulty = routing signal) |\n| `route_mode` | `\"confidence\"` | `confidence` (escalate iff `<confidence> < tau`) or `explicit` (`<route>`) |\n| `confidence_first` | `false` | commit `<confidence>` (+`<route>`) **before** solving, not after (see below) |\n| `tau` | `0.5` | escalation threshold on confidence (confidence mode) |\n| `lam` | `0.5` | **cost–quality dial** — the over/under-confidence knob |\n| `c_base` | `0.0` | base cost (sunk; cancels in the decision) |\n| `c_frontier` | `1.0` | escalation cost — only `λ·c_frontier` matters |\n| `oracle_acc` | `1.0` | simulated frontier accuracy (calculator = 1.0) |\n| `credit_own_accuracy` | `false` | **credit-assignment fix** — reward `base_correct` always, not the teacher's outcome (see below) |\n| `escalation_credit` | `1.0` | judgment bonus for escalating while `base_correct=0` (used by `credit_own_accuracy` and `turn_credit`) |\n| `turn_credit` | `false` | **turn-level credit assignment** — two-turn `MultiTurnEnv`, independent judgment/answer advantages (see below) |\n| `dataset_source` | `\"synthetic\"` | `synthetic` arithmetic / `table` (real teacher) / **`countdown`** (see below) |\n| `weights` | proxy=1, rest 0 | override (do **not** weight canaries); ignored when `turn_credit=true` |\n\n## Countdown dataset (`dataset_source=\"countdown\"`)\n\nSwaps the base task from arithmetic/GSM8K to **Countdown**: combine the given\nnumbers, each exactly once, using `+ - * /` and parentheses, to reach a target\n(`<answer>EXPRESSION</answer>` instead of a bare integer). Composes with every\nother feature — `route_mode`, `confidence_first`, `credit_own_accuracy`,\n`turn_credit` all work unchanged; only the problem/prompt/verifier differ.\nAlways uses the simulated-oracle escalation fallback (`oracle_acc`/`c_frontier`)\n— there's no precomputed real-teacher table for Countdown.\n\n**Why:** on arithmetic/GSM8K, `base_correct` never improved through RL training\nthis sprint (flat or declining across every reward variant tried — H6, H7,\n`escalation_credit` sweeps). Countdown is the best-documented case of RLVR\ndriving small models from near-random to near-100% (TinyZero and reproductions)\n— it directly tests whether *this* training setup can raise `base_correct` at\nall, on a task where the literature says it should be possible. Model-size\ncaveat: TinyZero's own results show a 0.5B model failing to improve at all\nwhile 1.5B/3B do — `sprints/Llama-3.2-1B-Instruct` sits between those data\npoints, so this is a real empirical question, not a guaranteed win. Difficulty\n(`easy`/`medium`/`hard`) controls search-space size via number count (3 or 4)\nand range, generated **guaranteed solvable by construction** (verified via\nbrute-force search in `tests/smoke_test.py`).\n\n**A genuinely different reward-hacking surface.** The verifier checks a\n*structural* property (uses each given number exactly once, evaluates to the\ntarget) rather than equality to one fixed integer — `_check_countdown` in\n`cascade_router.py` — which is exploitable in ways single-answer arithmetic\nisn't (malformed-but-plausible expressions, reusing/dropping a number). Worth\nwatching whether this compounds with the escalation-reward hacking this sprint\nalready studies. Countdown/RLVR is also documented to invite length/search-\npadding hacking distinctly from arithmetic — prime-rl's trainer ships a\nbuilt-in `length_penalty` specifically for this (see `docs/algorithms.md` in\nthe `prime-rl` repo), worth reaching for if it shows up here.\n\n```bash\nprime eval run cascade-router -m meta-llama/Llama-3.2-1B-Instruct -a '{\"dataset_source\": \"countdown\"}'\n```\n\n## Confidence-timing ablation (`confidence_first`)\n\nThe base config elicits confidence **after** the model has already worked the\nproblem in the same completion — so `stated_confidence` is a *retrospective*\nread on the model's own reasoning/answer, not a test of whether the routing\nsignal is even elicitable before solving. `confidence_first=true` requires the\nmodel to commit to `<confidence>` (and `<route>`, in explicit mode) **before**\nany worked solution, i.e. a *prospective* \"feeling of knowing\" (cf. Kadavath\n2022) estimated from the problem statement alone.\n\nNothing else changes: same reward, same teacher table, same parsing (the\n`<answer>`/`<confidence>`/`<route>` regexes are order-agnostic), same offline\ncanaries. This isolates a single question — **does prospective confidence\nroute worse than retrospective confidence?** — as a direct, item-matched A/B:\n\n```bash\nprime eval run cascade-router -m meta-llama/Llama-3.2-1B-Instruct -a '{\"confidence_first\": true}'\n```\n\nOr for the hosted GRPO run, use `sprint-config-conffirst.toml` (identical to\n`sprint-config.toml` except `confidence_first = true`) and diff\n`analysis/routing_metrics.py` output between the two runs on `split=\"test\"`.\nExpect `confidence_auroc_vs_base` / `escalation_recall_on_hard` to be **worse**\nhere if the retrospective signal in the baseline is doing real work (the model\n\"checking its own work\" via the tokens it already emitted) rather than the\nrouting decision being learned independent of solve order.\n\n## Credit-assignment fix (`credit_own_accuracy`)\n\n**The bug.** By default, an escalated rollout's trained reward is `final_correct`\n— the TEACHER's correctness — entirely independent of the student's own parsed\nanswer (still generated in the same completion, since nothing is truncated). A\nwrong self-answer that got bailed out by escalation scores identically to a\nright one. Across the `rollouts_per_example` samples for one item, this means\nGRPO's group-relative advantage rewards *escalating itself*, while the specific\n(possibly low-quality) worked-solution tokens preceding it ride along for free\n— `base_correct` gets no gradient at all whenever the policy escalates. This is\na believable mechanism for `base_correct` degrading over training under an\nunder-confident policy (heavy escalation).\n\n**The fix.** `credit_own_accuracy=true` changes `routed_reward` to:\n\n```\nescalate:      base_correct − λ·cost + escalation_credit·(1 − base_correct)\nnot escalate:  base_correct − λ·cost                    (unchanged — already the case)\n```\n\nThe quality term is now *always* `base_correct`, so every rollout has a live\ngradient toward better answers. The bonus fires only for a **correct judgment\ncall** (escalated while `base_correct=0`) — it does NOT depend on whether the\nteacher subsequently succeeds (that stays a separate, purely diagnostic\n`final_correct`/`teacher_correct` signal). Wasteful escalation (`base_correct=1`\nbut escalated anyway) is unchanged: still penalized only via cost.\n\n**Important limitation.** This is still a `SingleTurnEnv` with one scalar\nreward for the whole completion — GRPO applies the same advantage to every\ntoken, confidence/route tokens *and* worked-solution tokens alike. The fix\nremoves the semantic bug (no wrong answer is ever scored as \"correct\"), but it\nis **not** true token/turn-level credit assignment between \"reward the\njudgment\" and \"reward the answer\" — that would need a genuinely segmented\nreward architecture (see open next step below).\n\n```bash\nprime eval run cascade-router -m meta-llama/Llama-3.2-1B-Instruct -a '{\"credit_own_accuracy\": true, \"escalation_credit\": 1.0}'\n```\n\nOr `sprint-config-lam09-creditfix.toml` for the hosted run, matched against\n`sprint-config-lam09.toml` (the already-running λ=0.9 baseline).\n\n## Turn-level credit assignment (`turn_credit`)\n\n`credit_own_accuracy` fixes the *semantics* of the reward, but it's still one\nscalar for the whole completion — GRPO applies the same advantage to every\ntoken, judgment and answer alike. `turn_credit=true` goes further: cascade-router\nbecomes a genuine two-turn `vf.MultiTurnEnv` —\n\n- **Turn 0** — the model states `<confidence>`/`<route>` only, WITHOUT solving.\n- **Turn 1** — \"now solve it,\" `<answer>` only. Always runs, regardless of the\n  turn-0 decision (nothing truncated — same principle as `credit_own_accuracy`).\n\n`TurnCreditRubric` (a custom `Rubric.score_group` override) computes **two\nindependent group-relative advantages** and stamps them onto their own\ntrajectory step directly:\n- **`judgment_reward`** (turn 0): was the escalate/answer decision objectively\n  correct per Chow's rule (`escalate ⇔ base_correct=0`), plus its cost. Never\n  depends on the specific answer text.\n- **`answer_reward`** (turn 1): `base_correct`, always — pure accuracy, fully\n  decoupled from the routing decision.\n\nThis relies on a real `verifiers` mechanism, not a workaround: `Rubric.score_group`\nonly broadcasts its group-mean-subtracted advantage onto a trajectory step\n`if t[\"advantage\"] is None` — a step that already has an advantage set is left\nalone. Verified locally against the actual installed `verifiers` package\n(`tests/test_turn_credit.py`) before writing this into the env, including that\n`MultiTurnEnv`'s automatic `RubricGroup` wrapping (it adds a `MultiTurnMonitorRubric`\nfor the `num_turns` metric) doesn't interfere.\n\n**That integration risk was real.** The first hosted run (`s4idnwz1n83772gxum2tyd8u`,\nv0.1.11) failed after 10 consecutive batches with zero trainable rollouts\n(`BackoffLimitExceeded`). Root cause, from the orchestrator log — `Env\nnekoeiha/cascade-router@0.1.11 ready: num_tasks=766 group_scoring=False` — the\nbase `Rubric.has_advantages` defaults to `False`, and `has_group_rewards` only\nchecks for special \"group-shaped\" functions (mine are all plain per-rollout\ncanaries). With both `False`, the orchestrator never called `score_group` at\nall — only per-rollout `score_rollout`, which runs just the weight-0\n`canary_funcs`, giving a CONSTANT `reward=0.0` for every rollout → the\n`zero_advantage` filter correctly dropped every batch (no variance to compute\nan advantage from). **Fix (v0.1.12):** `TurnCreditRubric` now overrides both\n`has_group_rewards` and `has_advantages` to `True`, confirmed locally to\npropagate correctly (`env.requires_group_rollouts` / `env.provides_advantages`\nboth `True`, matching the exact fields the failing log line reported).\nOverriding `score_group` alone is NOT sufficient — the rubric must also declare\nthat it does so.\n\n```bash\nprime eval run cascade-router -m meta-llama/Llama-3.2-1B-Instruct -a '{\"turn_credit\": true}'\n```\n\n## Positioning\n\n- A new axis vs. DCPO/RLCR: **calibration-as-an-action under a cost constraint**, not calibration as a passive metric. Escalation is \"abstention with a fallback and a price tag.\"\n- Grounded in a real deployed pattern (model routers / cascades; the Georgian × Sublime two-tier system) — and a real Georgian use case.\n- Detector framing preserved: routing-calibration is read from weight-0 canaries; never weight them.\n\n## Files\n- `cascade_router.py` — env (arithmetic + oracle, routing reward, canaries, `load_environment`; `TurnCreditEnv`/`TurnCreditRubric` for `turn_credit=true`)\n- `analysis/routing_metrics.py` — escalation recall / false-escalation / regret / AUROC\n- `docs/hypothesis.md` — pre-registered λ-sweep hypotheses + the routing dynamic-range gate\n- `tests/smoke_test.py` — stubbed-`verifiers` tests (reward/parsing logic, fast, no dependencies)\n- `tests/test_turn_credit.py` — REAL-`verifiers` tests for `turn_credit=true` (MultiTurnEnv/RubricGroup/score_group semantics the stub can't model); run with `.venv/bin/python3 tests/test_turn_credit.py`\n","encoding":"utf-8","truncated":false,"total_bytes":15690},"status":null}