{"data":{"kind":"file","path":"README.md","version_id":"rw9ekgir93odrabsblqadarg","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3460,"modified_at":"2026-07-21T13:13:15.135000","content_hash":"0a03296ef4067559ec91b0b0fad4fc907e9d03f0389c28e13e03ffd9bb90ef86"},"entries":[],"content":"# linear-algebra\n\nLinear-algebra evaluation with a **reward-hacking-robust, math-aware grader**.\n\n### Overview\n- **Environment ID**: `linear-algebra`\n- **Short description**: Core undergraduate linear algebra (determinant, rank, trace, transpose, dot product, matrix–vector and matrix–matrix products, inverse, linear-system solutions, eigenvalues, eigenvectors, squared norm, symmetry test) with a grader that resists reward hacking and respects real linear-algebra ambiguity.\n- **Tags**: math, linear-algebra, reward-hacking-robust, eval, train\n\nThe value here is the **grader**. Numeric-answer evals are easy to game, and\nlinear algebra adds genuine ambiguity a naive checker gets wrong. This grader\nhandles both.\n\n### Datasets\n- **Primary dataset**: 78 problems packaged as `problems.json`, generated by `generate_problems.py`.\n- **100% original & self-verifying**: each problem is generated from a randomly drawn integer matrix/vector and its answer is computed by `numpy`/`sympy` — never hand-typed. No textbook text, exercises, or figures are reproduced; the underlying concepts are public mathematics.\n- **Split sizes**: 78 total — 13 topics × 3 difficulties (easy/medium/hard) × 2 variants (26 easy / 26 medium / 26 hard). Filter with the `difficulty` arg. Topics: determinant, rank, trace, transpose, dot_product, matvec, matmul, inverse, solve_system, eigenvalue, eigenvector, norm_squared, is_symmetric.\n\n### Task\n- **Type**: single-turn\n- **Output format**: free reasoning, then a final line `ANSWER: <value>` where value is a scalar (`6`), vector (`[1, 0, -2]`), or matrix (`[[1, 2], [3, 4]]`).\n- **Rubric overview**:\n  - `correct_answer` (weight 1.0): 1.0 only for a typed, correct, non-gamed answer.\n  - `format_ok` (weight 0.0): metric only — whether the `ANSWER:` line was used.\n\n**Grader defenses** (each of which a naive grader fails):\n1. Format discipline — no `ANSWER:` line, no reward (no \"last number in text\" fallback).\n2. Anti-flooding — several different `ANSWER:` values score 0.\n3. Prompt-echo guard — a scalar answer that merely echoes a prompt number (not the gold) scores 0.\n\n**Math-aware checking**:\n- Typed comparison (a vector answer to a matrix question scores 0).\n- Eigenvector / null-space answers accepted up to a nonzero scalar (`direction_free`); fixed-solution vectors are not.\n- Per-problem relative tolerance.\n- Formatting robustness (found via live rollouts): markdown wrappers (`**6**`, `` `6` ``) are stripped and multi-line matrix answers are parsed by bracket balancing — without weakening the defenses above.\n\n`test_grader.py` covers 16 adversarial + formatting cases (`python test_grader.py`).\n\n### Quickstart\nRun an evaluation with default settings:\n\n```bash\nprime eval run linear-algebra\n```\n\nConfigure model and sampling:\n\n```bash\nprime eval run linear-algebra -m openai/gpt-4.1-mini -n 20 -r 3 -t 1024 -T 0.7\n```\n\nFilter difficulty:\n\n```bash\nprime eval run linear-algebra -a '{\"difficulty\": \"hard\"}'\n```\n\n### Taskset Config\n\n| Field | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `difficulty` | str \\| null | `null` | Keep only `easy` / `medium` / `hard`; `null` uses all 78. |\n\n### Metrics\n\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` | Main scalar reward (= `correct_answer`). |\n| `correct_answer` | 1.0 for a typed, correct, non-reward-hacked answer, else 0. |\n| `format_ok` | 1.0 if the model used the `ANSWER:` line (diagnostic, unweighted). |\n","encoding":"utf-8","truncated":false,"total_bytes":3460},"status":null}