{"data":{"kind":"file","path":"README.md","version_id":"xddc6edy99jd3i9fzyz70s2n","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":9411,"modified_at":"2026-07-13T10:44:19.980000","content_hash":"d428a46ddae8e061f8f6238d3f2835662a4d014b12c6f18c382cdf2c1f6cc509"},"entries":[],"content":"# isaf-extraction\n\nStructured extraction from ISAF (International Security Assistance Force) press\nreleases: read one release, return three fields as JSON.\n\n```json\n{\"province\": \"kunduz\", \"minkilled\": 0, \"mincaptured\": 3}\n```\n\nA single-turn environment for the [Prime Intellect Environments\nHub](https://docs.primeintellect.ai/tutorials-environments/install). The task is\nsmall and real: the documents are genuine 2008–2013 ISAF operational updates,\nand the gold labels come from a hand-cleaned dataset with an auditable cleaning\nhistory.\n\n## Task\n\nGiven the text of one press release, extract:\n\n- **`province`** — the Afghan province the incident occurred in (name only, e.g.\n  `helmand`), lowercased.\n- **`minkilled`** — the minimum number of people reported killed (integer).\n- **`mincaptured`** — the minimum number of people reported captured (integer).\n\nThe model sees a shared instruction plus the release text; it must return only\nthe JSON object (no prose, no code fences).\n\n## Dataset\n\n- Source: [`strickvl/isafpressreleases`](https://huggingface.co/datasets/strickvl/isafpressreleases),\n  pinned to the immutable **`v2`** tag (the cleaned rows as of 2026-06-27 — `v1`\n  marks the pre-clean 2024 state and must not be used).\n- Split shipped: `train`, filtered to `province_status == \"ok\"` →\n  **3,820 rows**. Rows whose province could not be pinned from the text\n  (`province_status == \"unknown\"`) are **excluded**.\n- Each row becomes one task: the release text as the `prompt`, and the gold\n  (`province_clean` / `minkilled` / `mincaptured`) as typed fields on the task's\n  data.\n\nThe dataset also carries a `test` split (724 rows) which this environment never\nloads. That is the author's sealed exam set, kept out of the environment on\npurpose so that a prompt or weight optimiser running against it *cannot* reach\nthose rows — structurally, not by good manners.\n\n## Grading\n\nThe reward is a **weighted sum** of four checks, priced from a measured error\nanalysis of a real model's mistakes on this task (counts priced hardest, JSON a\nsmall gate):\n\n| check | weight | scores |\n|---|---|---|\n| `valid_json` | 0.10 | 1.0 if the output parses as a JSON object, else 0.0 |\n| `province` | 0.20 | 1.0 exact · 0.5 known alias · 0.5 district of the right province · else 0.0 |\n| `minkilled` | 0.35 | 1.0 if the integer matches exactly, else 0.0 |\n| `mincaptured` | 0.35 | 1.0 if the integer matches exactly, else 0.0 |\n\n**This is a training reward, not an exact-match benchmark.** It deliberately\ngives *partial credit* for near-misses (a district that sits inside the right\nprovince; a known province spelling variant) so a policy has a gradient to climb\nduring RL. It is intentionally more lenient than a strict exact-match grader — if\nyou need a hard benchmark number, score with exact-match yourself and don't mix\nthe two. (Partial credit is signal for learning, not a leaderboard claim.)\n\nAlongside the reward, every run reports one **metric** — `truncated`, the share of\nreplies the model didn't finish because it hit a token cap. It changes no score.\nIt exists because a cut-off answer emits no closing brace, scores 0, and is\ntherefore indistinguishable *inside an average* from an answer that was simply\nwrong. Set your token cap too low and this environment will tell you, instead of\nquietly handing you a number that means something other than what you think.\n\nThe environment also ships a model-free **validation** pass (`validate\nisaf-extraction`), which checks every row is scoreable — non-negative counts, an\nactual press release to read — before you spend anything on a model.\n\n## Known limitations\n\n*(Honest by design — read these before trusting a number from this environment.)*\n\n- **The district→province map is a small hand-built table (7 entries), not\n  exhaustive.** Most real districts won't earn the 0.5 \"right province, named the\n  district\" credit yet. A later revision derives the full map from the dataset's\n  own `citydistrict` column. Some districts (e.g. Arghandab) genuinely exist in\n  more than one province, so district-alone can't always disambiguate.\n- **`unknown`-province rows are excluded.** Releases where the province isn't\n  determinable from the text are a real and interesting case (they'd test a\n  model's willingness to abstain), but \"unknown\" is a weak, gameable gold label,\n  so they're left out for now.\n- **The gold province is not always spelled the way the release spells it.** In 75\n  rows (2.0%) it doesn't appear literally in the text — usually a transliteration\n  difference (\"Nangahar\" in the release, `nangarhar` in the gold), sometimes\n  because the release names only a district. These rows are kept deliberately: the\n  district partial-credit rung exists precisely for the second case, and dropping\n  them would delete the rows the reward handles most carefully.\n- **Counts are scored as exact-integer match only** — no partial credit for\n  off-by-one, and the \"minimum reported\" convention means a release describing\n  \"several\" is labelled conservatively.\n- **Province partial credit ≠ the author's frozen eval grader.** The author keeps\n  a separate hash-sealed exact-match grader for reporting held-out numbers; this\n  environment's reward and that grader legitimately disagree (e.g. a district\n  scores 0.5 here, 0.0 there). Don't compare scores across the two.\n- **Scope:** single-turn, English, ISAF 2008–2013 operational-update style. Don't\n  assume it generalises to other conflict corpora or document styles.\n\n## `gepa-prompts/` (not used by the environment)\n\nTwo system prompts evolved by [GEPA](https://arxiv.org/abs/2507.19457) against\n`gpt-oss-20b`, kept here because they may be useful to a harness. **The environment\ndoes not use them** — it ships the plain instruction above.\n\nScored on 100 rows the prompt search never saw (paired, same rows per arm):\n\n| prompt | score |\n|---|---:|\n| the environment's own | 0.803 ± 0.054 |\n| `terra` | 0.968 ± 0.020 |\n| `sol` | 0.880 ± 0.050 |\n\nThe gain is real, but both prompts were evolved **for one model**. Putting either in\nthe environment would tilt a multi-model comparison toward models that resemble\n`gpt-oss-20b`, which is why they sit here rather than in the task. In v1 terms: a\nsystem prompt belongs to the harness that attempts the work, not to the work itself.\n\n## Versions\n\n| version | API | notes |\n|---|---|---|\n| **0.2.0** | Verifiers **v1** (`verifiers.v1`, a `Taskset`) | current |\n| 0.1.0 | Verifiers v0 (legacy `SingleTurnEnv` + `Rubric`) | `prime env pull strickvl/isaf-extraction@0.1.0` |\n\nThe port to v1 changed **no task and no reward**. The four weights and the\npartial-credit rungs are byte-for-byte the same, and the v0 reference cases are\nre-scored through the v1 code to prove it, so numbers measured under 0.1.0 remain\ncomparable with numbers measured under 0.2.0.\n\nWhat changed is ownership. Under v0 this environment was a `SingleTurnEnv`: it\nowned the dataset, the system prompt, the rubric **and** the loop that called the\nmodel. Under v1 it is a `Taskset`, which owns only the work — the rows and the\nscoring. The thing that *attempts* the work (the harness) is supplied at run time\nby whoever runs the environment, so the same tasks can be put to a plain chat\nloop, a coding agent, or anything else that produces a trace.\n\n## Usage\n\n```bash\nprime env install strickvl/isaf-extraction\n```\n\nCheck every row is scoreable, with no model calls and no cost:\n\n```bash\nvalidate isaf-extraction\n```\n\nRun an evaluation with an API model:\n\n```bash\neval isaf-extraction -m deepseek/deepseek-v4-flash -n 20 -r 3\n```\n\nRequires `verifiers >= 0.2.0`. For the legacy v0 code path\n(`vf.load_environment(\"isaf-extraction\")` returning a `SingleTurnEnv`), pull\n`strickvl/isaf-extraction@0.1.0` and pin `verifiers < 0.2.0`.\n\n## Provenance\n\nThis dataset has a real research pedigree. The ISAF press-release collection was\nassembled for the Afghanistan Analysts Network (AAN) special report **\"A Knock on\nthe Door: 22 Months of ISAF Press Releases\"** (Alex Strick van Linschoten & Felix\nKuehn, 12 October 2011), which analysed 3,771 releases from December 2009 to\nSeptember 2011 and found a steady month-on-month rise in reported kills and\ncaptures alongside inconsistent use of \"leader\" / \"facilitator\" labels. The report\nwas covered by the *Guardian* the same day. The dataset was compiled and published\nto Hugging Face subsequently.\n\n- AAN report: <https://www.afghanistan-analysts.org/en/special-reports/a-knock-on-the-door-22-months-of-isaf-press-releases/>\n  ([PDF](https://www.afghanistan-analysts.org/wp-content/uploads/downloads/2012/10/AAN_2011_ISAFPressReleases.pdf))\n- *Guardian* coverage, 12 Oct 2011: <https://www.theguardian.com/world/2011/oct/12/nato-taliban-afghanistan-exaggerated>\n\nThe dataset itself was cleaned non-destructively (original columns preserved; a\nthree-tier fix taxonomy; \"unknown\" kept as an honest value; group-leakage handled\nacross splits). See the [dataset\ncard](https://huggingface.co/datasets/strickvl/isafpressreleases) for the full\ncleaning history and audit columns.\n\n*(Note: the AAN report analysed 3,771 releases over its 22-month window; this\nenvironment ships 3,820 `province_status=='ok'` rows from the cleaned `v2`\ndataset, so the two counts are related but not identical — the dataset is its own\ncompilation, not a 1:1 copy of the report's working set.)*\n\n## License\n\nApache-2.0.\n","encoding":"utf-8","truncated":false,"total_bytes":9411},"status":null}