{"data":{"kind":"file","path":"README.md","version_id":"i30zufx9n3xgz9ikl7fgdrfz","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":12827,"modified_at":"2026-09-12T11:34:52.200000","content_hash":"996c7576628e814869f51bd85de1735bad1e7293e4cc1964948355aec2066c7e"},"entries":[],"content":"# geo-reprojection-v1\n\nCoordinate reprojection and geodesy on the WGS84 / GDA2020 ellipsoid, graded against\n[PROJ](https://proj.org/) rather than a stored answer key.\n\n| | |\n|---|---|\n| **Archetype** | Deterministic reward, procedurally generated, `INFINITE` |\n| **Harness** | Any. Tool-less (`null`) makes it a numeric-reasoning task; `bash` makes it a tool-use task |\n| **Runtime** | No container (`NEEDS_CONTAINER` unset); run on a host `subprocess` runtime |\n| **Reward** | `within_tolerance` (weight 1.0): pass/fail against a physical tolerance; graded mode opt-in |\n\n## Task families\n\n| Family | Ask | Answer | Accepted within |\n|---|---|---|---|\n| `reproject` | WGS84 lon/lat → projected easting/northing | `easting, northing` in metres | 1 m, Euclidean in the destination grid |\n| `inverse` | Projected easting/northing → WGS84 lon/lat | `longitude, latitude` in decimal degrees | 1 m, measured as geodesic distance on the ellipsoid |\n| `distance` | Geodesic distance between A and B | metres | 10 m |\n| `bearing` | Initial forward azimuth from A to B | degrees clockwise from true north | 0.05°, modulo 360 |\n\nNine projected CRSs, each sampled only inside its true area of use: GDA2020 MGA zones\n54–56, WGS84 UTM zones 10N/33N/56S, British National Grid, NZTM2000, and Web Mercator.\nCoordinates are printed to six decimal places (degrees) or three (metres).\n\nEach prompt states the task, the units and order of the answer, the tolerance it is\ngraded against (interpolated from the configured value), and the parsing rule below.\nNothing in the prompt prescribes a method. British National Grid prompts additionally\nname the OSGB36 datum shift the key uses (see *Datum transformations*).\n\n## Answer format and parsing\n\nThe reply must contain an `<answer>...</answer>` block. The parser, as implemented:\n\n- grades the **last** `<answer>` block in the reply and ignores text outside it;\n- reads numbers separated by commas or whitespace; because commas are separators,\n  `3,941,601.087` is three numbers and a reply using thousands separators is\n  *unparseable* (`answered` 0, reward 0), not merely wrong;\n- requires exactly the number of values the family asks for — two for `reproject` and\n  `inverse`, one for `distance` and `bearing` — and treats any other count as unparseable;\n- accepts plain decimals and scientific notation, with any number of decimal places;\n- compares bearings circularly, so 359.99 and −0.01 are the same heading and any\n  equivalent angle modulo 360 is accepted.\n\n## Rewards and metrics\n\n| Name | Kind (weight) | Definition |\n|---|---|---|\n| `within_tolerance` | reward (1.0) | 1.0 iff the error is at most the row's tolerance, else 0.0; 0.0 when unparseable. With `task.graded = true`: `1 / (1 + (error / (tolerance × graded_halflife))²)`, a smooth decay still anchored on the tolerance |\n| `error` | metric | Physical error: metres for `reproject` (Euclidean in the destination grid), `inverse` (geodesic distance on the WGS84 ellipsoid) and `distance`; degrees for `bearing`. −1.0 when unparseable. A reply PROJ cannot evaluate — a latitude outside ±90°, which has no point on the ellipsoid, or an overflowing literal — reads as the finite sentinel `20003931.5` m (half the meridional circumference; `180.0`° for bearings), never NaN |\n| `error_ratio` | metric | `error / tolerance`, so 1.0 is the pass boundary in every family and the value is comparable across families where `error` is not. −1.0 when unparseable |\n| `answered` | metric | 1.0 when the reply parsed. Separates a wrong answer from no answer — they need different fixes |\n\n`error` mixes metres and degrees when families are mixed, so its run-level mean is not\nmeaningful across families; read `error_ratio`, or `error` per family.\n\n## Arguments\n\nSet on the taskset with `--env.taskset.<flag>`, or under `[env.taskset]` in a config toml.\nTask-level flags take `--env.taskset.task.<flag>` / `[env.taskset.task]`.\n\n| Flag | Default | Meaning |\n|---|---|---|\n| `--env.taskset.families` | `[\"reproject\", \"inverse\", \"distance\", \"bearing\"]` | Families to draw from; each generated row picks one uniformly |\n| `--env.taskset.position-tolerance-m` | `1.0` | Pass threshold for `reproject` and `inverse` rows, in metres |\n| `--env.taskset.distance-tolerance-m` | `10.0` | Pass threshold for `distance` rows, in metres |\n| `--env.taskset.bearing-tolerance-deg` | `0.05` | Pass threshold for `bearing` rows, in degrees |\n| `--env.taskset.seed` | `20260826` | Seeds the coordinate stream; the same seed regenerates the same tasks |\n| `--env.taskset.task.graded` | `false` | Fold the error curve into the reward instead of a hard pass/fail |\n| `--env.taskset.task.graded-halflife` | `1.0` | Error, in tolerance units, at which the graded reward decays to 0.5 |\n\nThe tolerances are quoted in the prompt text, so changing one also changes what the\nmodel is told. The taskset is infinite; bound every run with `-n`.\n\n## Dependencies\n\n`verifiers>=0.3.1` and `pyproj>=3.6`, which bundles PROJ and its EPSG database. No API\nkeys or environment variables, no network access, no PROJ grid files, and no container\n(`NEEDS_CONTAINER` unset).\n\n## Datum transformations\n\n\"WGS 84\" in the answer key means one named EPSG operation per grid, pinned in the\ntaskset (`PINNED_DATUM_SHIFT`) rather than left to PROJ's best-available choice, so the\nkey for a given seed does not depend on which grids the grading host has installed or\non `PROJ_NETWORK`:\n\n| Grid | Datum shift to WGS 84 | EPSG-stated accuracy |\n|---|---|---|\n| GDA2020 MGA zones 54–56 | `GDA2020 to WGS 84 (2)` — null transformation; the two datums are treated as coincident | 3 m |\n| NZTM2000 | `NZGD2000 to WGS 84 (1)` — null transformation | 1 m |\n| British National Grid | `OSGB36 to WGS 84 (6)` — the EPSG 7-parameter Helmert, **not** the OSTN15 grid; the prompt says so on these rows | 2 m |\n| WGS 84 UTM zones, Web Mercator | none — the projection is exact | — |\n\nThe reference answer is therefore \"what PROJ returns under that named operation\", and the\n1 m position tolerance is a tolerance against that reference, not against ground truth:\nthe EPSG operations themselves are only accurate to 1–3 m, and a solver that applies\nOSTN15 to a British National Grid row can land metres from the key.\n\n## Design notes\n\n**Why binary reward.** Partial credit on a numeric answer is a reward-hacking surface: a\nmodel that echoes the input, or emits a number of roughly the right magnitude, would bank\na fraction for free. The reward is pass/fail; the continuous error is recorded as a\n*metric* so a training run still has a dense signal to inspect. `--env.taskset.task.graded true`\nfolds the error curve into the reward when a run explicitly wants shaping.\n\n**Why tolerance is physical.** Errors are compared in the destination grid's own\nprojected metres: ground metres to within the 0.9996 scale factor on the transverse\nMercator and UTM grids, and up to about 2.9× ground metres for Web Mercator at the ±70°\nedge of its sampling box (one grid metre there is 1/cos 70° ground metres). Inverse rows\nare compared as geodesic distance on the WGS84 ellipsoid, so a longitude error costs the\nsame near the pole as at the equator. A tolerance on the printed decimals would instead\nbe far stricter for a 7-digit easting than for a 3-digit longitude.\n\n**Why procedural.** Coordinates are drawn to six decimal places inside each CRS's area of\nuse, so the stream never repeats and there is no answer key to leak. `INFINITE = True`;\nbound every run with `-n`.\n\n**Why the key is re-derived in `validate()`.** Every row's expected answer is recomputed\nfrom the inputs the model reads — the same PROJ calls, applied to the shipped numbers —\nand every coordinate is round-tripped forward and back through PROJ. A generator bug\nsurfaces as a key/input disagreement, and a coordinate drawn outside its CRS's usable\ndomain fails the round-trip, before a model ever sees the row.\n\n## Run\n\n```bash\n# Model-free: re-derive every key from its shipped inputs and round-trip through PROJ\nuv run validate geo-reprojection-v1 -n 40 --runtime.type subprocess\n\n# The measured run, with every flag spelled out (the repository's\n# configs/geo-reprojection.toml sets the model, 40 tasks, temperature 0 and the null\n# harness, but not the subprocess runtime or the episode timeout)\nuv run eval geo-reprojection-v1 -n 40 --model openai/gpt-5-mini \\\n  --env.agent.harness.id null --env.agent.runtime.type subprocess \\\n  --env.timeout.episode 480 --sampling.temperature 0\n\n# One family at a tighter tolerance\nuv run eval geo-reprojection-v1 -n 20 --model <model-id> \\\n  --env.agent.harness.id null --env.agent.runtime.type subprocess \\\n  --env.taskset.families '[\"reproject\"]' --env.taskset.position-tolerance-m 0.5\n```\n\nThe flags matter for a tool-less task: without `--env.agent.harness.id null` verifiers\nresolves the agent to the `bash` harness, and without `--env.agent.runtime.type\nsubprocess` even the null harness provisions a Prime sandbox for every rollout although\nnothing here needs a container. `--env.timeout.episode 480` bounds a rollout that gets\nstuck in provider retries; the measured run below had no episode timeout and hung.\n\n## Status\n\n| Check | Result |\n|---|---|\n| `uv run validate geo-reprojection-v1 -n 40 --runtime.type subprocess` | 40/40 valid — every key re-derived from its shipped inputs, every coordinate round-tripped within 1 cm |\n| Offline test suite (pytest, in the repository) | Passes. Asserts that PROJ's own answer scores 1.0 in every family, and that echoing the input, omitting the `<answer>` block, scattering candidate numbers in prose, using thousands separators, supplying the wrong number of values, or answering with a latitude off the ellipsoid all score 0.0 (in graded mode too); that `validate()` rejects a tampered key in every family; and that the datum shift is pinned |\n| Eval — `openai/gpt-5-mini` via Prime Inference, 2026-09-11 | 40 tasks requested, all four families, temperature 0.0, `null` harness, `subprocess` runtime: 36 traces recorded (the run was cut after 4 rollouts hung in a provider-500 retry loop with no timeout), 34 `agent_completed`, 2 provider errors excluded. `within_tolerance` 0.0882 (3/34); `answered` 1.000; `error` median 92.2, min 0.0, max 4,538,768.7, deciles [0.1, 0.7, 9.9, 52.4, 92.2, 114.4, 957.9, 2686.4, 7265.4] — `error` mixes metres and degrees across families, so the deciles are the summary, not a mean |\n| Eval — `openai/gpt-5-mini`, 2026-09-12, revised prompt | Same 40 tasks and settings plus `--env.timeout.episode 480`: 37/40 completed, 3 hit the episode timeout (recorded as errors, no trace). `within_tolerance` **0.135** (5/37); `answered` 0.946; `error` median 39.1 |\n| Eval — `openai/gpt-5-nano`, 2026-09-12 | 40/40 completed. `within_tolerance` **0.100** (4/40); `answered` 0.950; `error` median 122.9 |\n| Eval — `deepseek/deepseek-v4-flash`, 2026-09-12 | 37/40 completed, 3 episode timeouts. `within_tolerance` **0.081** (3/37); `answered` 1.000; `error` median 4,778.7 |\n| Eval — `google/gemini-2.5-flash-lite`, 2026-09-12 | 40/40 completed. `within_tolerance` **0.000** (0/40); `answered` 0.975; `error` median 208,230.9 |\n| Reading across models | Every model answers in the required format almost every time, and every model is far outside the tolerances almost every time — the ordering (gpt-5-mini > gpt-5-nano > deepseek-v4-flash > gemini-2.5-flash-lite) is visible in the `error` medians long before it shows in the binary reward, which is what the graded metric is for. The `error` medians mix metres and degrees across families |\n| `bash` harness (tool use) | Not yet run against a model |\n| `task.graded = true` | Not yet run against a model |\n\nReading of the measured run: gpt-5-mini produces a well-formed answer on every task but\nlands far outside the 1 m / 10 m / 0.05° tolerances; the reward separates \"wrong\" from\n\"unparseable\" exactly as designed.\n\nCaveats:\n\n- Prime Inference returned bursts of upstream 500s on gpt-5-mini during that session and\n  the run had no episode timeout, which is why it was cut at 36 traces;\n  `--env.timeout.episode 480` was adopted for the later runs in the same session and is\n  part of the command above.\n- The prompt wording was revised after that run: the graded tolerance, the \"last block\n  is graded\" rule and \"no thousands separators\" are now stated, the bearing prompt reads\n  \"from A to B\", and British National Grid prompts name the OSGB36 datum shift. The\n  reward, the parser, the tolerances and the answer keys are unchanged, so the numbers\n  above are a record of the earlier prompt.\n- `error_ratio` was added after the 2026-09-11 run; the 2026-09-12 runs record it (see the uploaded evaluations), and it is not summarised above because its mean is dominated by a few catastrophic answers.\n","encoding":"utf-8","truncated":false,"total_bytes":12827},"status":null}