{"data":{"kind":"file","path":"README.md","version_id":"tcfjf8gf7427fgadaz6fdztj","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":28319,"modified_at":"2026-08-11T23:10:10.985000","content_hash":"cd46ca9c71e09fe9394d1dc146c93fa63034a55ca0c1a5a09ba5bb10dca56b35"},"entries":[],"content":"# octave-rl\n\nA seeded GNU Octave coding curriculum for native `verifiers.v1`. Every task is\na function signature plus a natural-language spec; scoring runs the submitted\nfunction against six hidden NumPy-derived cases on a pinned Octave interpreter\nand awards the fraction that pass. No judge model, no partial credit for\nlooking plausible, and the pass count is computed host-side from values the\ncandidate process never receives, so printing cannot raise a score.\n\nThe task pool is generated rather than scraped, so it cannot leak from a public\nbenchmark and its hidden test cases are unbounded. Ten families times eight\nvariants times three levels gives **240 distinct prompts**; it is still not an\nunbounded supply of distinct problems, and a seed split holds out test inputs\nrather than questions — see \"500 tasks: Questions vs Prompts\" before reporting a\nnumber.\n\n## Contents\n\n| Section | In one line |\n|---|---|\n| [Model naming conventions](#model-naming-conventions) | The full slugs behind \"Nemotron\" and \"Qwen XB\" referenced throughout. |\n| [Quickstart](#quickstart) | Install, and the two lines that load the environment. |\n| [Choosing a runtime](#choosing-a-runtime) | Where candidate code executes — Prime Sandbox, local subprocess, or a pinned rootfs — and when to use each. |\n| &nbsp;&nbsp;&rarr; [Scoring boundary](#scoring-boundary) | What the interpreter running candidate code can and cannot see. |\n| [What this actually tests](#what-this-actually-tests) | A failure taxonomy: this is a language-fluency benchmark far more than a reasoning one. |\n| [Families, levels, and difficulty](#families-levels-and-difficulty) | The ten task families, what each one exercises, and measured per-family pass rates. |\n| [500 tasks: Questions vs Prompts](#500-tasks-questions-vs-prompts) | The pool carries 240 distinct prompts; a held-out seed still holds out inputs, not questions. |\n| &nbsp;&nbsp;&rarr; [Two ways to hold out a problem](#two-ways-to-hold-out-a-problem) | The `families` and `variants` fields, and which question each answers. |\n| [Reward](#reward) | Fraction of hidden cases passed, the attempt discount, and the diagnostic metrics reported alongside. |\n| [Output shape is graded](#output-shape-is-graded) | Orientation counts, every prompt states the shape it will be compared against, and why. |\n| [Multi-turn and the optional guide](#multi-turn-and-the-optional-guide) | The retry loop, the LLM guide, the discount table, and the credential the guide needs on disk. |\n| [Multi-turn scoring](#multi-turn-scoring) | Most of the gap between a 1-turn and a 3-turn number is resampling, not capability. |\n| [Configuration](#configuration) | Every taskset and user field, with defaults. |\n| [Reproducibility](#reproducibility) | The pinned interpreter, seeds, and what makes two runs comparable. |\n| [Changes in 0.4.0 …](#changes-in-040) | Version history, newest first. |\n\n## Model naming conventions\n\nTwo model families appear throughout, as reference points and as the guide.\nNamed in full once here, and by short name everywhere after.\n\n| Short name | Full model id | Role |\n|---|---|---|\n| **Nemotron** | `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16` | Strong reference. Never trained here — it is an instrument for measuring the *taskset*. |\n| **Qwen 4B** | `Qwen/Qwen3.5-4B` | The original training target; has since outgrown Level 1. |\n| **Qwen 2B** | `Qwen/Qwen3.5-2B` | Baselined, not trained. |\n| **Qwen 0.8B** | `Qwen/Qwen3.5-0.8B` | The current training target. |\n| **Qwen 35B-A3B** | `Qwen/Qwen3.5-35B-A3B` | The optional retry guide, not a subject of any measurement. |\n\n**Every Qwen here is a Qwen3.5**, so after this point they are written\n**Qwen *size*** — `Qwen 4B`, `Qwen 0.8B`. A bare \"Qwen\" with no size is\nambiguous between four models and should be treated as an error in the text.\n\"Nemotron\" is unambiguous because only one is used.\n\n## Quickstart\n\n```bash\nprime env install kaiser-factorial/octave-rl\n```\n\n```python\nimport verifiers as vf\n\nenv = vf.load_environment(\"octave-rl\", level=1, num_tasks=200, seed=0)\n```\n\nCandidate code has to run somewhere. Pick a runtime before you evaluate — see\nbelow; the default reaches for Prime Sandboxes.\n\n## Choosing a runtime\n\n| `octave_runtime` | Where candidate code runs | Use for |\n| --- | --- | --- |\n| `\"prime\"` (default) | one short-lived Prime Sandbox per execution | runs whose numbers you report |\n| `\"local\"` | a bounded subprocess on the calling host | development, CI, pod-local training |\n\nBoth use the same input-only runner and the same host-side scorer, so hidden\nvalues never enter the interpreter that runs model output either way. They\ndiffer in containment and in speed: local scoring of the full 1,500-task pool\ntakes minutes, against roughly five minutes of cold provisioning *per\ncandidate* through a Sandbox.\n\nThe local runtime scores on whatever Octave it is pointed at, so pin it:\n\n```bash\nexport OCTAVE_RL_OCTAVE_ROOTFS=/opt/octave-rootfs   # unpacked gnuoctave/octave:10.2.0\n# or, if Octave is already installed:\nexport OCTAVE_RL_OCTAVE_BIN=/usr/local/bin/octave-cli\n```\n\nWith a rootfs, candidates run under `unshare --net` → `chroot` → `ulimit`\nbounds and see neither the host filesystem nor a network. If a network\nnamespace cannot be obtained the backend refuses to run unless\n`OCTAVE_RL_ALLOW_UNISOLATED_LOCAL=1` is set, and records then report\n`network_isolated = false`. `unshare`/`chroot` are Linux-only; on macOS run the\nwhole thing in the pinned image instead:\n\n```bash\ndocker run --rm --platform linux/amd64 -v \"$PWD\":/w -w /w \\\n  -e OCTAVE_RL_OCTAVE_BIN=/usr/local/bin/octave-cli \\\n  -e OCTAVE_RL_ALLOW_UNISOLATED_LOCAL=1 \\\n  gnuoctave/octave:10.2.0 <your command>\n```\n\n### Scoring boundary\n\nThe candidate sandbox receives only the candidate function, a generated\ninput-only runner, and the hidden *inputs*. It serialises each result's shape\nand flattened values into a terminal `__OCTAVE_CANDIDATE_RESULT__<token>`\nrecord carrying a token minted after generation. The trusted Python process\nkeeps expected outputs and pass counters entirely outside that sandbox,\nvalidates the record, and applies the shape and tolerance comparison itself.\n\nCandidate stdout therefore cannot set a score: printing a plausible result\nrecord fails the token check, and the counters it would need to forge are never\nin its process. Scoring does not use exit status. The in-Octave comparison\nharness and its `__OCTAVE_HARNESS_RESULT__` protocol exist only in the separate\nreference-pool validator, where the executed function is repository-owned.\n\n`TaskData.image`, workdir, and container resources are deliberately unset: the\nouter harness and user simulator run in their worker subprocesses, and only\ncandidate execution creates the pinned Octave Sandbox. This avoids a duplicate\ncontainer without weakening the boundary.\n\n## What this actually tests\n\nWorth knowing before you train on it. A failure taxonomy over 222 baseline\nrollouts put the competencies in this order:\n\n1. **can the model emit runnable Octave at all** — ~36% of rollouts fail here;\n2. **does it follow Octave's shape and orientation conventions** — ~5%;\n3. **does it obey the prompt's stated constraints** (e.g. no loops) — ~4%;\n4. **is the algorithm correct** — ~3%.\n\nThis is a language-fluency and convention-compliance benchmark far more than a\nmathematical-reasoning one. That makes it a good RL substrate: the reward is\ndeterministic, costs a few milliseconds, and is computed from hidden values the\ncandidate never sees, which removes the usual reward-hacking surface. It does\nnot make the reward unhackable — a family whose hidden cases are satisfiable by\na degenerate answer would still be gamed, which is what\n`scripts/validate_natural_solutions.py` exists to catch. A result obtained here\nis a result about learning a language's surface conventions.\n\n## Families, levels, and difficulty\n\nTen families cover reductions, logical indexing, reshape/permutation, broadcast\narithmetic, sliding windows, linear solves, recurrences, matrix min/max\nwrangling, string parsing, and signal identities.\n\n- **Level 1** — diagnostic scalar/vector and simple matrix functions.\n- **Level 2** — the working set, with more constraints and edge cases.\n- **Level 3** — level 2 plus a no-loops constraint, or a harder variant.\n\n**Family matters about 7x more than level.** Measured per-family pass rates\nspan a far wider range than the level ladder does, so a nominal \"level 2 mix\"\ncan contain anything from a near-unsolvable family to one the model passes\nthree times in four. If you are choosing a training mix, choose it on family\ncomposition and target families near p = 0.5, where sampled groups carry the\nmost GRPO advantage. Do not choose it on level alone.\n\nPass rates at T=1.0, 72-96 rollouts per family, thinking off, single turn,\nseed `20260808`, on the current taskset. Treat these as a difficulty ordering\nfor two specific models, not as a property of the tasks. What these numbers\nwere before the 0.2.0 repair, and how far each family moved, is in\n\"Changes in 0.2.0\".\n\n| family | Nemotron | Qwen 4B | note |\n|---|---:|---:|---|\n| `linsolve_tolerance` | 0.752 | 0.292 | |\n| `struct_cell_wrangle` | 0.736 | 0.389 | matrix min/max; the name is historical |\n| `logical_index` | 0.639 | 0.312 | |\n| `sequence_recurrence` | 0.574 | 0.366 | |\n| `reshape_permute` | 0.569 | 0.292 | |\n| `broadcast_arith` | 0.458 | 0.083 | |\n| `reduce_along_dim` | 0.389 | 0.326 | |\n| `string_parse` | 0.366 | 0.056 | |\n| `signal_identity` | 0.234 | 0.086 | FFT circular autocorrelation |\n| `sliding_window` | 0.213 | 0.190 | stride semantics |\n\n**Why this matters for training.** A family no model ever passes contributes\nzero GRPO advantage at any group size, so a taskset with a dead family wastes\npart of every rollout budget spent on it. Every family here is reachable: the\nlowest pass rate is **0.213** (Nemotron) and **0.056** (Qwen 4B). At\n`group_size = 8`, unanimous — therefore gradient-free — groups run at **0.100**\non Nemotron Level 2 and **0.143** on Qwen 4B Level 2.\n\n## 500 tasks: Questions vs Prompts\n\nRead this before reporting a number from this environment.\n\nA prompt is the signature plus a **variant** description plus a generated shape\nsentence. A variant is a named choice — which statistic, which axis, which\noperator — that changes what the function must compute, not merely which numbers\ntest it. Each of the ten families declares **eight**, rendered at three levels,\nfor **240 distinct prompts**.\n\n**Before 0.5.0 that number was 30**, one per (family, level), and every task in a\ncell shared a byte-identical prompt. That is the largest change in this\nenvironment's history, and no measurement taken before it is comparable at task\nor family level.\n\n### What a seed still does not do\n\nPools are separated by seed, and by task the separation is exact — training seed\n`0` and held-out seed `20260808` share 0 of 1,500 tasks. **They still share all\n240 prompts.**\n\nThat is not a defect and no variant count fixes it: with eight variants and about\nfifty tasks per family, every variant appears in every 500-task pool with\nprobability ≈ 0.999, so two seeds contain the same prompt set whatever the\nselection rule. A seed holds out hidden test *inputs*, never the question.\n\nSo `num_tasks = 500` is 500 test-suite draws over 240 problems rather than 30 —\nmuch better, and still in-distribution on the problem. **To hold out a problem,\nuse one of the two fields below**, and quote generalization from those rather\nthan from a seed split.\n\n### Two ways to hold out a problem\n\n| field | costs | holds out | use when |\n|---|---|---|---|\n| `families` | a fifth of training coverage | whole families — an unpracticed problem *type* | testing transfer to an idiom never trained |\n| `variants` | nothing; every family stays in training | a quarter of the problems, inside families the model trains on | testing whether a practiced idiom generalizes across its parameters |\n\n**The variant holdout is the stricter test and the cheaper one**, and it exists\nonly because prompts are parameterised. Names are `\"family:key\"`:\n\n```toml\n[taskset]\nvariants = [\"reduce_along_dim:mean-rows\", \"reduce_along_dim:median-columns\"]\n```\n\n`DEFAULT_HELDOUT_VARIANTS` holds out two per family;\n`specs.complement(declared_variants(), DEFAULT_HELDOUT_VARIANTS)` returns the\ntrained remainder. It is **frozen**, and chosen from measurement rather than\nposition: variants are ranked by measured solve rate, the middle half of each\nfamily's range is kept so the test sits on neither floor nor ceiling, and the\npair differing in the most spec components is taken so the test crosses a\nfamily's dimensions instead of running along one.\n\n| family | held out |\n|---|---|\n| `broadcast_arith` | `product`, `min` |\n| `linsolve_tolerance` | `solutionresidual-overdetermined`, `residualvector-overdetermined` |\n| `logical_index` | `even-extract`, `magnitude-zero` |\n| `reduce_along_dim` | `mean-columns`, `range-columns` |\n| `reshape_permute` | `perm213-row`, `perm312-column` |\n| `sequence_recurrence` | `order1-total`, `order2-terms` |\n| `signal_identity` | `shift-forward`, `autocorr-linear` |\n| `sliding_window` | `mean-stride1`, `range-strided` |\n| `string_parse` | `semicolon-integers-column`, `mixed-decimals-row` |\n| `struct_cell_wrangle` | `sumcount-columns`, `minmedmax-rows` |\n\nChanging this list changes what \"held out\" means in any published number, so\ntreat a change as a version bump rather than a tweak.\n\nThe family holdout is unchanged: `DEFAULT_HELDOUT_FAMILIES` is\n`[\"reduce_along_dim\", \"reshape_permute\"]` and `training_families()` returns the\ncomplement. Both sit mid-difficulty, so neither is floored nor ceilinged;\n`reduce_along_dim` has a near neighbour that stays in training\n(`struct_cell_wrangle` is also a column-wise reduction) so it tests transfer of a\npracticed idiom, while `reshape_permute` has none.\n\nBoth filters *select from* the full ten-family stream rather than cycling over\nyour selection, so a family's k-th task is byte-identical whichever families and\nwhichever variants are present. A train split and a holdout split drawn from one\nseed are therefore disjoint **and** each individually comparable to a full-pool\nmeasurement. Task ids come from the full stream, so they are stable but not\ncontiguous within a filtered pool.\n\n**Use three splits, and do not blend them.**\n\n| split | families | seed | used for |\n|---|---|---|---|\n| train | the trained ones | training | rollouts and gradient |\n| validation | **the same trained ones** | held-out | level promotion, checkpoint selection |\n| test | **the held-out ones** | held-out | generalization — read rarely, ideally once |\n\nTwo rules that are easy to get wrong:\n\n- **Report the splits as separate numbers, never as one weighted score.**\n  Averaging across families is what concealed a family stuck at 0.030 and a\n  level stuck at 0.000 in this environment for weeks; a blended score rebuilds\n  exactly that blindness.\n- **Never gate promotion or select checkpoints on the held-out families.** That\n  is selection leakage: you would be tuning against the thing you are claiming\n  is untouched. Use the validation split for every decision, and keep the test\n  split for the final read.\n\nAlso measure the held-out families *before* training. Base rates run from 0.21\nto 0.75 across families, so an absolute post-training score is uninterpretable;\nthe quantity that means something is the change from base on the same config.\n\n## Reward\n\n`case_fraction` — the fraction of hidden cases passed — is the only reward.\nThere is no bonus for execution, for formatting, or for a\ncandidate-controlled result report. Formatting, vectorization, execution rate\nand transposition are recorded as **metrics**, not folded into the score:\n\n| metric | meaning |\n| --- | --- |\n| `execution_fraction` | cases that ran without raising |\n| `correct_given_executed` | of those, the fraction correct |\n| `transposed_fraction` | non-passing cases whose result is exactly the transpose of the expected value |\n| `format_ok` | exactly one fenced block was emitted |\n| `vectorized` | no `for`/`while` in the submission |\n\n`execution_fraction` and `correct_given_executed` are worth logging separately\nduring training: they separate \"cannot write Octave\" from \"wrote the wrong\nalgorithm\", and those move independently.\n\n## Output shape is graded\n\nScoring compares `size(actual)` against the expected value's shape exactly, so\norientation is part of every task — a correct answer returned as a column when\na row was asked for scores zero. Because of that, **every prompt states its\nexpected output shape**, in a sentence generated from the same expected values\nthe grader compares against:\n\n```\nWrite this GNU Octave function:\n\n    function out = linsolve_tolerance(A, b)\n\nSolve the square linear system A*x=b.\nReturn a column vector (N-by-1).\nReturn exactly one fenced `octave` code block. Hidden tests include edge cases.\n```\n\nArguments arrive as their signatures imply — a matrix as a matrix, a column\nvector as a column — so the natural solution is conformant as written.\n`scripts/validate_natural_solutions.py` in the source repository enforces this\nby running a deliberately naive solution per family and level and requiring it\nto pass.\n\n## Multi-turn and the optional guide\n\n`max_turns` maps to the persistent user's attempt budget. The user simulator\nruns the submitted function after each attempt and reports back:\n\n| Attempt | Feedback | Correctness multiplier |\n| --- | --- | ---: |\n| 1 | none | 1.00 |\n| 2 | hidden pass count and Octave diagnostic | 0.85 |\n| 3 | the same diagnostic plus one concise guide hint | 0.60 |\n\nThe guide defaults to Qwen 35B-A3B (`Qwen/Qwen3.5-35B-A3B`) through Prime\nInference. It sees\nthe public task prompt, the candidate source, and the first diagnostic only —\nnever hidden inputs, expected values, or the reference implementation.\nCredentials are never written into traces or package configuration. Set\n`guide_enabled = false` for a fully self-contained run with no external calls.\n\n> **Credential note — read this before enabling the guide.** The user simulator\n> runs in its **own subprocess**, and `PRIME_API_KEY` **is not inherited by\n> it**. Exporting the variable in the shell that launches training or eval is\n> not enough. Write the credential where the subprocess can read it:\n>\n> ```bash\n> mkdir -p ~/.prime && printf '{\"api_key\": \"%s\"}\\n' \"$PRIME_API_KEY\" > ~/.prime/config.json\n> ```\n>\n> `prime login` produces that file already, so an interactively authenticated\n> machine works out of the box; a pod or sandbox that only has the environment\n> variable does not. Measured on 2026-08-09: with the credential unreachable,\n> the guide turn failed on every third attempt, which cost 20–33% of rollouts\n> in a training run.\n>\n> Since 0.3.1 a guide failure **degrades to an unguided retry** instead of\n> ending the rollout, logs a warning, and records the reason in\n> `state.guide_unavailable`. So a misconfigured run now loses hints rather than\n> rollouts — but it is still misconfigured, and the hints are what attempt 3 is\n> for.\n\n## Multi-turn scoring\n\nMeasured on 2026-08-09 and worth knowing before you report anything from a\nmulti-turn configuration.\n\n**Retries are the strongest lever in the environment.** Solve rate from one turn\nto three: Nemotron 0.570 → 0.828 (L1), Qwen 4B 0.332 → 0.715. Gains of\n+0.22 to +0.38, larger for the weaker model.\n\n**But almost none of it comes from the feedback.** A control that replaced the\nwhole diagnostic with the sentence \"That answer was not correct.\" kept\n**79–95%** of the gain; the informative version beat it by +0.031/+0.016/+0.008,\nnone distinguishable from zero. Two-turn solve also lands *below* independent\nresampling (0.688 against the 0.815 that `1−(1−p)²` predicts), which is what\ncorrelated attempts look like.\n\nSo **the multi-turn scaffold is approximately correlated best-of-N sampling**\nwith a small informational bonus. The one component with a measured effect is\nthe LLM guide — a specific diagnosis of the actual bug — worth **+0.062 at\nLevel 1** (t = 2.37) on top of the extra attempt.\n\nTwo rules follow:\n\n- **Always state the turn budget with a score.** A 3-turn number is not\n  comparable to a 1-turn number, and most of the gap is resampling.\n- **Measure with `raw_case_fraction`, not the reward.** `case_fraction`\n  multiplies correctness by an attempt discount (0.85 on attempt 2, 0.60 on\n  attempt 3), so thresholding it cannot count a success after the first attempt.\n  The two coincide only at one turn.\n\n## Configuration\n\n`level`, `num_tasks`, `seed`, `require_vectorized`, `families` and `variants`\nare typed Taskset config fields; `octave_runtime`, `max_attempts` and\n`guide_enabled` are task and user fields.\n\n```toml\n[taskset]\nlevel = 2\nnum_tasks = 256\nseed = 20260808\n# Both holdout fields default to None, meaning \"everything\". See\n# \"Two ways to hold out a problem\" before setting either.\nfamilies = [\"logical_index\", \"broadcast_arith\"]          # whole families\nvariants = [\"reduce_along_dim:mean-rows\"]                # \"family:key\" pairs\n\n[taskset.task]\noctave_runtime = \"local\"\n\n[taskset.task.user]\noctave_runtime = \"local\"\nmax_attempts = 1\nguide_enabled = false\n```\n\nThe package exports both `OctaveTaskset` and a `load_environment(level=1,\nnum_tasks=500, max_turns=2, require_vectorized=False, seed=0, **kwargs)` entry\npoint. It is native `verifiers.v1` throughout and does not mix in the legacy v0\n`vf.Environment` form.\n\n## Reproducibility\n\n- Interpreter image: `gnuoctave/octave:10.2.0` (Docker Hub)\n- Observed interpreter: GNU Octave 10.2.0\n- Verifiers: `>=0.2.1,<0.3` · NumPy: `>=2.0,<3`\n- Each task is fully determined by `(level, seed, task index)`.\n- Six hidden cases per task, generated with NumPy and stored with the task.\n\n## Changes in 0.5.0\n\n**Breaking change to task semantics. No measurement taken before this version is\ncomparable at task or family level.**\n\n- **Parameterised descriptions.** Each family declares eight **variants** — a\n  named choice of statistic, axis or operator that changes what the function must\n  compute. The pool goes from **30 distinct prompts to 240**. Prompt, reference\n  and naive solution are written together from one definition per variant, so\n  they cannot drift.\n- **A variant holdout**, `variants`, alongside the family holdout. It holds out\n  problems inside families the model still trains on, so it costs no training\n  coverage. `DEFAULT_HELDOUT_VARIANTS` is frozen and chosen from measured\n  per-variant solve rates.\n- **`solved`**, an undiscounted 0/1 metric, so solve rate never has to be\n  recovered by thresholding a reward that is discounted by attempt.\n- **`validate_natural_solutions.py` checks per variant.** It previously held one\n  naive solution per (family, level), which would have covered one variant of\n  eight and reported PASS for the other seven.\n- **`audit_constant_outputs.py` checks per element**, within tolerance and\n  anchored from both ends of variable-length outputs. It previously asked only\n  whether a whole output was constant, and therefore could not see a single\n  graded position that never varies — which 0.4.x `linsolve_tolerance` level 3\n  had, at 1.33e-14 against a 1e-7 tolerance.\n- **`reshape_permute` prompts tightened** after measurement showed they induced\n  runaway generation: 25–61% of rollouts hit the completion cap against 0–11%\n  elsewhere. The disambiguation that prevents the inverse permutation reading is\n  unchanged and was re-verified.\n- The ten pre-variant generator functions and the `DESCRIPTIONS` table are\n  **deleted** rather than left beside the live code.\n\nGates on this pool, on the pinned Octave 10.2.0: naive solution **9,000/9,000**\nwith every variant checked by its own, reference path **9,000/9,000**, and no\nlevel-1 variant near zero for both Qwen3.5-4B and Nemotron.\n\n## Changes in 0.4.1 and 0.4.2\n\nDocumentation only — no change to task generation, scoring, or any config\nfield, so numbers measured on 0.4.0 remain comparable.\n\n- The quickstart named `<owner>` instead of `kaiser-factorial`, so the first\n  command a reader copies did not work.\n- Added a Contents table and a model-naming block: \"Qwen\" alone was ambiguous\n  between four models, and every Qwen here is a Qwen3.5, so they are written\n  Qwen *size* throughout.\n- Reordered the sections to follow use: run it, then what it measures, then how\n  it scores, then reference.\n- Two claims were stronger than the code supports and are now hedged with the\n  mechanism and the residual failure mode. The reward is not unhackable: a\n  family whose hidden cases were satisfiable by a degenerate answer would still\n  be gamed, which is what `scripts/validate_natural_solutions.py` exists to\n  catch.\n- The per-family table now states current difficulty only; the 0.1.0 → 0.2.0\n  movement it used to carry is in \"Changes in 0.2.0\".\n\n## Changes in 0.4.0\n\n- The retry feedback is a composed diagnostic instead of raw Octave stdout. The\n  old message was 46% `__OCTAVE_CANDIDATE_RESULT__` transport blob by length,\n  repeated each identical error once per case with a random temp path, and for\n  33% of retries carried no diagnostic at all. It now strips the blob and paths,\n  deduplicates errors, and names which failure mode occurred: did not run, ran\n  with the wrong shape (stating both shapes), or ran with the right shape and\n  wrong values. A six-case syntax failure went from 1,084 characters to 110.\n- The guide fires **on need** — the first retry whose diagnostic cannot help —\n  rather than at a fixed attempt number. 31% of rollouts reach their first retry\n  with no execution error, where the hint is the only thing that can help.\n- The attempt discount follows the hint rather than the attempt number: a hinted\n  solve is priced at `guided_attempt_multiplier` whenever the hint arrived.\n- No score change is expected from any of this; see \"Multi-turn scoring\"\n  means\".\n\n## Changes in 0.3.1\n\n- A guide failure no longer ends the rollout. It previously raised out of\n  `respond`, and the MCP layer turned that into a contentless tool result the\n  host reported as `JSONDecodeError('Expecting value: line 1 column 1')` --\n  naming neither the cause nor this package. It now degrades to an unguided\n  retry, warns, and records the reason in `state.guide_unavailable`.\n- Documented that `PRIME_API_KEY` is **not** inherited by the user-simulator\n  subprocess; use `~/.prime/config.json`. See the credential note above.\n\n## Changes in 0.3.0\n\n- `families` is now a taskset config field, so train and eval pools can be\n  disjoint by *problem* rather than only by hidden inputs. See \"Holding out a\n  family gives you a real one\" above. Backward compatible: omitting it selects\n  all ten families and reproduces 0.2.x exactly.\n\n## Changes in 0.2.0\n\nTask semantics changed; **0.2.0 scores are not comparable to 0.1.0 scores at\nthe task level.** Family names, task ids, level structure, hidden expected\nvalues and reward multipliers are unchanged, so family-level comparison holds.\n\n- `linsolve_tolerance` now receives `b` as a column, so `A\\b` is conformant.\n  Previously it arrived as a row and every hidden case failed with\n  `nonconformant arguments` no matter what the model wrote — the family scored\n  0.030 and 0.000 for two different models.\n- `broadcast_arith` now receives `a` as a column, matching its own prompt, so\n  `a + b` broadcasts as written.\n- Every prompt states its graded output shape. The previous blanket \"Preserve\n  input orientation\" line described neither the rule being enforced nor the\n  output, and is gone.\n- `reshape_permute` levels 2–3 describe their real contract. They previously\n  said \"For a 3-D array A\" against a signature taking a flat vector and a size\n  triple; both models scored `correct_given_executed = 0.000` across 96\n  rollouts.\n- Every level-3 description restates its own task. Two of them had been\n  compressed to the point of dropping the specification —\n  `struct_cell_wrangle` fell from 0.792 at level 2 to 0.000 at level 3 on the\n  same computation.\n- A generation truncated inside its code fence no longer reaches Octave with\n  the opening fence attached, which had turned every such case into an\n  uninformative line-1 syntax error.\n","encoding":"utf-8","truncated":false,"total_bytes":28319},"status":null}