{"data":{"kind":"file","path":"README.md","version_id":"jwa77ixq8rxzgrzngtibkwxx","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":12011,"modified_at":"2026-08-01T19:10:59.335000","content_hash":"65c77a782ba58beb8dfa368f44a3bc389a80550ea7c3fabc9453429e85c854fb"},"entries":[],"content":"# scaling-book\n\nA single-turn `verifiers` environment built from [How To Scale Your Model](https://jax-ml.github.io/scaling-book/), the free systems textbook from Google DeepMind on the arithmetic of training and serving large models: rooflines, arithmetic intensity, collectives, sharding strategies, and inference economics.\n\n117 items, every one traceable to a specific chapter. 72 of them are validated against executable reference solutions before the dataset is allowed to build.\n\nThe reason to care about this one rather than another textbook scrape is the last section of this README: it ships a held-out transfer check, and the held-out half is not a random 20% holdout. It is the same physics on different hardware.\n\n## Install and run\n\n```bash\nuv run vf-install scaling-book\nuv run vf-eval scaling-book -m gpt-4.1-mini -n 20\n```\n\nOr from a clone:\n\n```bash\npython data/build_dataset.py          # regenerate and revalidate the JSONL\npython tests/test_graders.py          # offline grader tests, no API key needed\npython tests/test_reward_hacking.py   # adversarial tests, no API key needed\npython tests/test_transfer_check.py   # reporting arithmetic, no API key needed\npython transfer_check.py --model gpt-4.1-mini\n```\n\n## Configs\n\n| config | contents | n |\n|---|---|---|\n| `qa` | chapters 1-8, question-answer | 69 |\n| `code` | chapters 1-8, implementation | 20 |\n| `train` | `qa` + `code`, the whole in-distribution half | 89 |\n| `heldout_qa` | chapter 12, question-answer, GPU hardware | 24 |\n| `heldout_code` | chapter 12, implementation, GPU hardware | 4 |\n| `heldout` | both held-out splits | 28 |\n| `all` | everything | 117 |\n\n`load_environment` also takes `eval_fraction` (default 0.2, assigned by a stable hash of the item id so the split does not move between runs or machines), `code_timeout` (default 20s), and `allow_code_execution` (default `True`).\n\n**Train on `train`. Do not train on `heldout`.** Doing so does not break anything; it just destroys the only measurement in here that is hard to fake.\n\n## Task types and grading\n\nFive grader types, dispatched per item off `info[\"grader\"]`:\n\n**numeric** — one number. Graded on relative error against a per-item tolerance, with partial credit: full marks inside the tolerance, 0.4 within 3x the tolerance (capped at 10% relative error, so a loose per-item tolerance cannot widen the near-miss band indefinitely), 0.1 for landing within a factor of two, zero otherwise. Wrong sign is wrong, not nearly right. The bands are deliberately tight, and `tests/test_reward_hacking.py` grid-searches constant answers to check that they stay that way. The tolerance is per-item because \"how many chips do you need\" wants an exact integer and \"how long does this step take\" does not. Answer extraction strips unit words so `240` and `240 FLOPs/byte` grade the same, but deliberately does *not* strip magnitude prefixes — `0.24 kFLOPs/byte` is answering a different question than the one that was asked, and the system prompt says so.\n\n**numeric_multi** — several numbers in a stated order, scored as the mean of the per-number credits. This is what makes \"give the parameter count and the FLOPs and the step time\" a single item instead of three, and it means a model that gets two of three right is visibly better than one that gets none.\n\n**mcq** — single letter, binary.\n\n**formula** — a symbolic expression. Graded by `sympy` equivalence rather than string match, so `(1/8)*T/D` and `T/(8*D)` both pass. Inequalities are handled by splitting on the relation, checking the operator matches (with both orientations treated as the same claim, so `B/X > R` and `R < B/X` agree), and comparing the difference of the two sides — so `T > 8*D` does not silently equal `T < 8*D`. Each item can also carry hand-written `equivalents` for forms sympy will not unify. The answer is model-authored text going into a computer-algebra parser, so it passes a whitelist first — length cap, character class, and every name in the expression restricted to that item's declared symbols plus a fixed function set — and is then parsed with `parse_expr` under a minimal namespace rather than `sympify`, which evaluates.\n\n**code** — one Python function, executed in a subprocess against test cases, scored as the fraction of tests passed *above the trivial baseline*. The baseline is what a submission that ignores its arguments and returns a constant can pass, measured per item at build time; it is subtracted, and the remainder rescaled, so a knowledge-free submission scores exactly zero and the reference solution still scores exactly one. Without that subtraction, roughly half the implementation items paid a third of full marks for `return 0`, because a degenerate test case's answer really is zero. See the warning below.\n\nThe rubric weights `correctness` at 1.0 and a format reward at 0.2. The format reward is the parser's, rescaled: the parser's own version has a floor of 0.2 for arbitrary text, which at weight 0.2 would pay a policy 17% of the maximum for producing nothing at all. Subtracting the floor makes it a bonus for formatting rather than a participation fee. What is left — 0.2 for any well-formed response, whatever it says — is intentional, and under a group-relative algorithm it contributes no advantage, since every rollout in the group that formats correctly earns it equally. `exact` (binary) and `heldout_flag` are registered as metrics at weight zero, so they show up in reports without moving the reward.\n\n## Provenance\n\nEvery item declares where it came from, and the mix is reported at build time:\n\n- `book_stated_fact` (64) — the answer is stated in the book. Hardware specs, roofline thresholds, the constants in the tables.\n- `book_worked_problem` (48) — the book poses the problem and works it. These are the ones where the reasoning chain matters.\n- `synthesized_from_book` (5) — assembled from book material into a question the book does not literally ask, but whose answer follows from what it states.\n\nThe taxonomy exists so that anyone can tell which items would survive the book being rewritten and which would not. Chapter distribution: ch1 8, ch2 12, ch3 13, ch4 17, ch5 11, ch6 9, ch7 11, ch8 8, ch12 28.\n\n## The validation gate\n\n`data/build_dataset.py` refuses to emit any JSONL unless all of the following hold:\n\n1. Every item id is unique.\n2. Every item declares a provenance from the allowed set.\n3. Every numeric target is finite and every tolerance is in (0, 1).\n4. Every multiple-choice answer letter is actually among that item's choices.\n5. Every symbolic answer and every declared equivalent parses under sympy.\n6. **Every implementation task has a reference solution, and that reference passes every one of that task's test cases.**\n7. No knowledge-free constant passes all of an implementation task's tests, and whatever fraction the best one does pass is recorded on the item so the grader can subtract it.\n\nCheck 6 is the one that matters. 24 reference solutions, 72 test cases, all executed at build time. They live in `data/reference_solutions.py` and are written in a deliberately different style from the task docstrings, so passing is a statement about the tests rather than about copying. Without this, a test-case suite is a guess about a test-case suite, and a wrong test case is worse than a missing one — it teaches the policy to be confidently wrong and pays it for the privilege.\n\n`tests/test_graders.py` covers the other direction: that each grader pays out for a correct answer and refuses to pay out for a wrong one, including a code stub that defines the right function name and returns garbage, and one end-to-end rollout scored through the rubric itself rather than by calling the reward functions directly.\n\n`tests/test_reward_hacking.py` covers the direction that actually bites. It grid-searches a knowledge-free policy over constant answers and fails the build if the best one clears 0.05 mean correctness (currently 0.034, from constants that happen to be right); it runs every code exploit found in review — reading the answer key, rewriting the answer key, speaking the grader's stdout protocol directly and claiming 100,000 passes, returning a truthy non-bool to a predicate task — and requires all of them to score exactly zero; and it checks that a symbolic answer cannot execute, by asserting that a payload which used to rewrite `_numeric_credit` in the grading process leaves it identical.\n\n## Held-out transfer check\n\nChapters 1 through 8 of the book work through TPUs: a specific set of FLOPs and bandwidth numbers, and a torus interconnect where the cost of a collective depends on how many hops around the ring. Chapter 12 works the identical reasoning on GPUs, where every hardware constant is different and the interconnect is a switched fat tree rather than a torus.\n\nSo chapter 12 is a natural experiment that a random holdout cannot give you. The method transfers — roofline, arithmetic intensity, when a collective stops being free, how to pick a sharding — and none of the numbers do.\n\n```bash\npython transfer_check.py --model gpt-4.1-mini --rollouts 4\n```\n\nIt evaluates a model on `train` and on `heldout` and prints the gap in points on the binary `exact` metric, plus a per-grader breakdown and a matched-grader gap that reweights the held-out half to the in-distribution grader mix. That last number exists because the two halves do not have identical proportions of code and multiple-choice items, and an unadjusted gap partly measures composition rather than transfer.\n\nA policy that learned the method holds its accuracy across the two halves. A policy that memorised 240 and 2550 does not, and the gap widens over training even while in-distribution reward climbs. That divergence is the signal this environment is built to expose.\n\n## Warning: code execution\n\nWith `allow_code_execution=True` (the default), implementation tasks execute model-authored Python. Here is exactly what that means, because \"runs in a subprocess\" is not a security property.\n\nThe submission runs in a throwaway temporary directory as `python -I -S`, with a minimal environment, stdin closed, in its own process group, under `RLIMIT_AS` 512MB, `RLIMIT_CPU` 30s, `RLIMIT_NOFILE` 64, `RLIMIT_NPROC` 64 and `RLIMIT_FSIZE` 8MB, with a wall-clock timeout (`code_timeout`, default 20s) after which the whole process group is killed rather than just the direct child.\n\nThe part that matters more than the limits is what the child is given. **It never sees an expected value or a tolerance.** It receives arguments only, in `inputs.json`, and returns computed values on a nonce-delimited line of stdout; the parent holds the entire answer key and does every comparison in its own process. Result lists are truncated to the number of test cases before scoring. So the ways a submission could previously buy a passing grade — reading the test file that used to sit in its own working directory, rewriting it with `tol=1e9`, or printing a hundred thousand successes and exiting — now buy nothing. Forging a pass requires producing the correct number, which is the task. `tests/test_reward_hacking.py` runs each of those attempts as an assertion.\n\nWhat this is still not: a sandbox. The child runs as the same user with the same filesystem and the same network. A submission can read your home directory and open a socket. The design goal here was that a reward-maximising policy gains nothing by trying, not that a deliberately malicious one is contained. Run it on a machine you control, on a model you are training. Anywhere else, pass `allow_code_execution=False` — implementation tasks then score zero and no subprocess is spawned — or run the whole thing in a container.\n\n## Source and license\n\nContent is derived from *How To Scale Your Model* (Google DeepMind), which is publicly available at https://jax-ml.github.io/scaling-book/. Items are questions and worked problems built from the book's material, not reproductions of its text.\n","encoding":"utf-8","truncated":false,"total_bytes":12011},"status":null}