{"data":{"kind":"file","path":"README.md","version_id":"tfi0n4do6asslg3pmklz8w9m","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6007,"modified_at":"2026-07-11T16:15:46.671000","content_hash":"589180f25e6a3615412f447e2dffb5f7c12c3bb9f0a1d198f90080879c337fd8"},"entries":[],"content":"# isaf-extraction\n\nStructured extraction from ISAF (International Security Assistance Force) press\nreleases: read one release, return three fields as JSON.\n\n```json\n{\"province\": \"kunduz\", \"minkilled\": 0, \"mincaptured\": 3}\n```\n\nA single-turn environment for the [Prime Intellect Environments\nHub](https://docs.primeintellect.ai/tutorials-environments/install). The task is\nsmall and real: the documents are genuine 2008–2013 ISAF operational updates,\nand the gold labels come from a hand-cleaned dataset with an auditable cleaning\nhistory.\n\n## Task\n\nGiven the text of one press release, extract:\n\n- **`province`** — the Afghan province the incident occurred in (name only, e.g.\n  `helmand`), lowercased.\n- **`minkilled`** — the minimum number of people reported killed (integer).\n- **`mincaptured`** — the minimum number of people reported captured (integer).\n\nThe model sees a shared instruction plus the release text; it must return only\nthe JSON object (no prose, no code fences).\n\n## Dataset\n\n- Source: [`strickvl/isafpressreleases`](https://huggingface.co/datasets/strickvl/isafpressreleases),\n  pinned to the immutable **`v2`** tag (the cleaned rows as of 2026-06-27 — `v1`\n  marks the pre-clean 2024 state and must not be used).\n- Split shipped: `train`, filtered to `province_status == \"ok\"` →\n  **3,820 rows**. Rows whose province could not be pinned from the text\n  (`province_status == \"unknown\"`) are **excluded** from v0.1.\n- Each row becomes a `question` (instruction + release text) and an `answer`\n  (the gold JSON, from `province_clean` / `minkilled` / `mincaptured`).\n\n## Grading\n\nThe reward is a **weighted sum** of four checks, priced from a measured error\nanalysis of a real model's mistakes on this task (counts priced hardest, JSON a\nsmall gate):\n\n| check | weight | scores |\n|---|---|---|\n| `valid_json` | 0.10 | 1.0 if the output parses as a JSON object, else 0.0 |\n| `province` | 0.20 | 1.0 exact · 0.5 known alias · 0.5 district of the right province · else 0.0 |\n| `minkilled` | 0.35 | 1.0 if the integer matches exactly, else 0.0 |\n| `mincaptured` | 0.35 | 1.0 if the integer matches exactly, else 0.0 |\n\n**This is a training reward, not an exact-match benchmark.** It deliberately\ngives *partial credit* for near-misses (a district that sits inside the right\nprovince; a known province spelling variant) so a policy has a gradient to climb\nduring RL. It is intentionally more lenient than a strict exact-match grader — if\nyou need a hard benchmark number, score with exact-match yourself and don't mix\nthe two. (Partial credit is signal for learning, not a leaderboard claim.)\n\n## Known limitations\n\n*(Honest by design — read these before trusting a number from this environment.)*\n\n- **The district→province map is a small hand-built table (7 entries), not\n  exhaustive.** Most real districts won't earn the 0.5 \"right province, named the\n  district\" credit yet. A later revision derives the full map from the dataset's\n  own `citydistrict` column. Some districts (e.g. Arghandab) genuinely exist in\n  more than one province, so district-alone can't always disambiguate.\n- **`unknown`-province rows are excluded** in v0.1. Releases where the province\n  isn't determinable from the text are a real and interesting case (they'd test a\n  model's willingness to abstain), but \"unknown\" is a weak, gameable gold label,\n  so it's deferred to a possible v0.2.\n- **Counts are scored as exact-integer match only** — no partial credit for\n  off-by-one, and the \"minimum reported\" convention means a release describing\n  \"several\" is labelled conservatively.\n- **Province partial credit ≠ the author's frozen eval grader.** The author keeps\n  a separate hash-sealed exact-match grader for reporting held-out numbers; this\n  environment's reward and that grader legitimately disagree (e.g. a district\n  scores 0.5 here, 0.0 there). Don't compare scores across the two.\n- **Scope:** single-turn, English, ISAF 2008–2013 operational-update style. Don't\n  assume it generalises to other conflict corpora or document styles.\n\n## Usage\n\n```bash\nprime env install strickvl/isaf-extraction\n```\n\n```python\nimport verifiers as vf\n\nenv = vf.load_environment(\"isaf-extraction\")   # returns a SingleTurnEnv\n# then hand `env` to an evaluator or an RL trainer\n```\n\nQuick local evaluation with an API model:\n\n```bash\nvf-eval isaf-extraction -m gpt-5-nano -n 20 -r 3\n```\n\n## Provenance\n\nThis dataset has a real research pedigree. The ISAF press-release collection was\nassembled for the Afghanistan Analysts Network (AAN) special report **\"A Knock on\nthe Door: 22 Months of ISAF Press Releases\"** (Alex Strick van Linschoten & Felix\nKuehn, 12 October 2011), which analysed 3,771 releases from December 2009 to\nSeptember 2011 and found a steady month-on-month rise in reported kills and\ncaptures alongside inconsistent use of \"leader\" / \"facilitator\" labels. The report\nwas covered by the *Guardian* the same day. The dataset was compiled and published\nto Hugging Face subsequently.\n\n- AAN report: <https://www.afghanistan-analysts.org/en/special-reports/a-knock-on-the-door-22-months-of-isaf-press-releases/>\n  ([PDF](https://www.afghanistan-analysts.org/wp-content/uploads/downloads/2012/10/AAN_2011_ISAFPressReleases.pdf))\n- *Guardian* coverage, 12 Oct 2011: <https://www.theguardian.com/world/2011/oct/12/nato-taliban-afghanistan-exaggerated>\n\nThe dataset itself was cleaned non-destructively (original columns preserved; a\nthree-tier fix taxonomy; \"unknown\" kept as an honest value; group-leakage handled\nacross splits). See the [dataset\ncard](https://huggingface.co/datasets/strickvl/isafpressreleases) for the full\ncleaning history and audit columns.\n\n*(Note: the AAN report analysed 3,771 releases over its 22-month window; this\nenvironment ships 3,820 `province_status=='ok'` rows from the cleaned `v2`\ndataset, so the two counts are related but not identical — the dataset is its own\ncompilation, not a 1:1 copy of the report's working set.)*\n\n## License\n\nApache-2.0.\n","encoding":"utf-8","truncated":false,"total_bytes":6007},"status":null}