{"data":{"kind":"file","path":"README.md","version_id":"ydt8j16arsk24cw3tv2ry19x","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5461,"modified_at":"2026-10-28T06:43:49.923000","content_hash":"61b11ac26e455645808d44043924bd28062b6ef5fb79657a3e2dbc1760975f63"},"entries":[],"content":"# Paper Ablation Master v0.3 — Research Methods Benchmark\n\n**Scope:** standalone, host-scored benchmark core. This is a real, executable research-protocol testbed, **not yet a complete Prime Intellect v1 Hub taskset or a genuine paper-reproduction benchmark**. In particular, candidate artifacts are regression coefficients; evaluating arbitrary agent-written training programs requires an external sandbox executor which has NOT been implemented in this version. Earlier v0.2 code is not silently treated as compatible with this release.\n\n## New in v0.3\n\n- **24 deterministic tasks** across six generalization regimes and four seeds. Each task has independently derived train / development / hidden datasets.\n- **Host-owned hidden evaluation**: final reward uses a secret test split not surfaced in development feedback. This limits direct score leakage; it is *not* cryptographic protection from an attacker with direct Python import access to Task.data or this package.\n- **Phase-gated research campaign**: reproduction -> single-variable ablation -> improvement. Mutable experiments are snapshots tied to code digests and a per-session run budget.\n- **Integrity-aware results**: HMAC-signed experiment receipts (signed by the evaluator, not by the candidate), hash-linked experiment journal with replay/corruption checks, final report signature.\n- **Visible experiment cost** and a deterministic suite manifest for paired task comparisons.\n- **Reward components**: reproduction 25%, effect of controlled ablation 20%, hidden improvement 30%, cited evidence 10%, development/hidden generalization 10%, report completion 5%.\n- **Reference solver**: ridge regression and robust iteratively reweighted regression, executable as a local demo.\n\n## Installation\n\n```bash\npython -m pip install -e '.[dev]'\npytest -q\npam-demo --family robust_contamination --seed 17\npython -m paper_ablation_master.cli --family correlated_features --seed 11\n```\n\nRun programmatically:\n\n```python\nfrom paper_ablation_master import Task, Campaign, reference_candidate\nfrom paper_ablation_master.suite import Suite\nprint(Suite().manifest()) # 24 tasks\nc=Campaign(Task('robust_contamination',17),budget=18)\nprint(c.protocol())\n# call c.run('reproduction', ...), c.run('ablation', ...), ...\n```\n\n## Recommended agent workflow\n\n1. Read public task and paper-like hypothesis sheet. Generate a research plan and persist into `Journal`.\n2. Train a baseline in an external **untrusted** container; emit only the trained model artifact plus code digest/config. Never give container access to private holdouts, secret, or grader code.\n3. Controlled ablation (exactly one declared variable changes; same code hash).\n4. Improvement branch (new code hash allowed); save experiments, failure notes, and checkpoints across agent restarts.\n5. Submit 3 signed receipts and a claim. Trusted host scores hidden data once, finalizes the campaign and records results.\n6. Report mean reward, experiment cost and family coverage across full 24-task suite.\n\n## Security boundaries and known limitations\n\n- Standalone test mode imports the same `Task` class used for private data generation. **This is not safe as an exposed evaluator inside the agent runtime**; production deploy must put scorer in an isolated service and never ship hidden-generation details to the agent.\n- Code digests are self-declared in this standalone build; a real executor must hash the files it executes and bind that verified digest to its signed receipt.\n- The scorer checks the declared ablation control, **not a full provenance diff across every hyperparameter**; harden with exact config diff plus sandbox-run provenance before deployment.\n- Experiment receipts do not bind resource use, hardware, wall-time, GPU consumption or stdout; production grading needs immutable execution attestations.\n- Synthetic tasks mimic research methodology. No claims to match any published-paper reported metric, and no publisher paper benchmark is included.\n- The score for ablation rewards the magnitude of a controlled effect, not agreement to a published ablation result. It therefore needs reference-result alignment before training use.\n- The simplistic report score checks length, not scientific truthfulness. Judge/structured-calibration scoring requires further work.\n- No hosted model rollout, Docker/E2E run, v1 taskset packaging, Hub acceptance or bounty eligibility has been verified.\n\n## v1 Hub integration path\n\nCurrent official Prime `verifiers` docs describe **Taskset + Harness + Agent runtime** (not legacy `ToolEnv`). Scaffold using `uv run vf-init paper-ablation-master -T`, put the trusted host scoring behind a task lifecycle/toolset, use a supported MCP harness and runtime, and verify the adapter against a pinned `verifiers` release. This repository intentionally does **not** claim to export a working v1 adapter without a dependency-bound validation run.\n\nDocs: https://github.com/PrimeIntellect-ai/verifiers/blob/main/docs/v1/tasksets.md\n\n## Quality gates to complete before submission\n\n- External sandbox integration with signed host execution receipts + artifact store\n- Hidden data generation in a separate trusted service\n- Real licensed open paper implementations with reproducibility targets and citations\n- Statistical multi-seed confidence intervals, compute-normalized efficiency, and adversarial shortcut tests\n- Taskset v1 install/validate/eval with real model; baseline and stronger agent separation\n","encoding":"utf-8","truncated":false,"total_bytes":5461},"status":null}