{"data":{"kind":"file","path":"README.md","version_id":"levahqw8ev6oyjijdcs0xnoa","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":4198,"modified_at":"2026-09-13T04:06:06.937000","content_hash":"9e1bfc443fee29d5fc2d652d418cf185b6b96252f1854f2d3a8fb14fda4b8e76"},"entries":[],"content":"# deshuffle-papers\n\nReconstruct complete research papers from shuffled paragraphs, solved by an agent in a sandbox: each task samples one or more full papers, mixes all of their paragraphs, and writes them to `paragraphs.jsonl` in the rollout workspace (one JSON line per paragraph: numeric `label` + `text`; displayed order may already be correct). The agent inspects the file with Bash or Python and answers with a `\\boxed{}` JSON permutation (one paper) or list of per-paper label lists (multiple papers). The reward is `grouping_f1 * norm_kendall_tau` — B-cubed F1 over paragraph-to-paper assignment times macro-averaged normalized within-paper pairwise concordance — so both correct paper grouping and correct internal ordering are required.\n\nEvery task sets `network_allow=[]`: the agent box gets framework-only egress, so the model can process the local file but cannot look the papers up online. This needs a network-policy runtime — `docker`, or `prime` with `vm=true`; `subprocess` is rejected (`NEEDS_CONTAINER`).\n\n## Taskset\n\n- **Source:** [PrimeIntellect/deshuffle-papers-v1-corpus](https://huggingface.co/datasets/PrimeIntellect/deshuffle-papers-v1-corpus), pinned to revision `dbd5cd49aff9fc1bbdc2579ff6d861ca2c868bfa` — 1,000 permissively licensed papers (500 arXiv, 500 bioRxiv; 983 CC BY 4.0, 17 CC0 1.0), non-body sections and poor-quality extractions filtered during preprocessing\n- **Size:** procedurally generated; the taskset is declared infinite and streams tasks lazily. Task `idx` anchors paper `idx % 1000` for corpus coverage; beyond 1000 the papers repeat with fresh shuffles and co-papers. Each task samples its remaining papers without replacement, with the paper count uniform in `min_papers..max_papers`. Generation is deterministic in `(idx, seed)`; training streams the taskset, eval runs must be bounded with `-n`\n\n## Response format\n\nOne paper — a flat JSON permutation of all labels:\n\n```text\n\\boxed{[3, 1, 4, 2]}\n```\n\nMultiple papers (the prompt states the exact count) — a JSON list of per-paper label lists, outer order irrelevant, every label exactly once:\n\n```text\n\\boxed{[[3, 1, 4], [7, 2], [5, 8, 6]]}\n```\n\nThe scorer accepts a different number of nonempty predicted groups, so merge and split mistakes receive partial grouping credit rather than becoming parse failures.\n\n## Scoring\n\n```text\nordering_reward = grouping_f1 * norm_kendall_tau\n```\n\n- **`grouping_f1`** — B-cubed F1 of predicted versus true paper membership; penalizes contaminated predicted groups and true papers split across groups.\n- **`norm_kendall_tau`** — macro-average normalized pairwise concordance within true papers; only pairs assigned to the same predicted group are comparable, a paper with no comparable pair scores 0, and a random internal ordering has expected value 0.5. For one-paper tasks the reward reduces to this term.\n- Metrics: `parsed_ok`, `grouping_precision`, `grouping_recall`, `grouping_f1`, `norm_kendall_tau`, and the diagnostic-only `paper_exact_match`. Invalid label partitions score 0 everywhere.\n\nScoring uses linear grouping statistics and merge-sort inversion counting (`O(N log N)`), so it scales to the entire corpus without materializing paragraph pairs.\n\n## Configuration\n\n- `min_papers` / `max_papers`: inclusive range of complete papers per task; both default to 1\n- `seed`: task-generation seed\n- `source`: optionally restrict papers to `arxiv` or `biorxiv`\n- `include_headers`: include section headings as orderable units\n\n## Changelog\n\n- 2026-08-31: Stream the downloaded JSONL into the retained sampling pool instead of materializing a second raw-record list.\n- 2026-08-17: Made the taskset an infinite lazy stream (`INFINITE`); removed `num_tasks` — bound eval runs with `-n`. Added config-field docstrings and narrowed `source` to a literal.\n- 2026-08-13: Blocked internet egress by default (`network_allow=[]`, `NEEDS_CONTAINER`); the agent must reorder the local file, not look papers up online.\n- 2026-08-12: Migrated to current v1 conventions — dropped the `_v1` suffix, moved the shuffled paragraphs onto wire-safe `TaskData`, replaced init-time checks with pydantic validators.\n- 2026-07-17: Initial v1 taskset.\n","encoding":"utf-8","truncated":false,"total_bytes":4198},"status":null}