{"data":{"kind":"file","path":"README.md","version_id":"jjb0ifovu13o78v56tvgvsnk","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3224,"modified_at":"2026-10-29T17:07:32.546000","content_hash":"e8ea5f667639d0e2ef3d206a30f0f61c34cfed77e6ca5aa664224fee905bb8df"},"entries":[],"content":"# binary-reasoning-v1 — open sample\n\n**Open sample: 100 of 10,251 verified reverse-engineering tasks**, plus all 29 agentic\nflag-hunt challenge binaries with sources. The full pack (10,251 items, same schema) is a\npaid product: complete-pack licensing with provenance, updates and grading material —\nask (samueljanttila@gmail.com), datasheet below.\n\nEvery item was produced by static analysis (Ghidra via REA, binutils) and machine-checked:\nanswers are derived from the artifact itself and never leak into the input\n(audited 0/10,251). Per-item provenance: binary path, sha256_16, address, arch.\n\n## What is in this sample (100 items, selected by published script)\n`select_sample.py` in this package reproduces the exact draw (seed 20261029, family quotas,\nall minority-architecture items included). The sample includes gold answers so you can check\nour graders before you buy.\n\n| family | in full pack | in sample | grading |\n|---|---|---|---|\n| func_id_exact (identify stripped/internal function from disassembly) | 9,073 | 40 (all non-x86-64 items included: AArch64 11, RISC-V 3, PPC64 3, MIPS 2, i386 1, ARM 1; rest x86-64) | deterministic symbol match (alias-tolerant) |\n| func_id_mc (multiple-choice identification) | 1,036 | 12 | deterministic option match |\n| agentic_flag_hunt (find argv[1] that prints OK; binaries in `bins/`) | 29 | 29 (whole family — this is the public benchmark) | exact flag / stdout + exit-code probe |\n| func_summary (summarize scrubbed function; rubric judge) | 53 | 6 | LLM judge + visible-constants rubric |\n| crossarch_equivalence (same function across two ISAs?) | 25 | 5 | deterministic stance+symbol |\n| library_ident (SBOM-style name+version from artifacts) | 14 | 3 | exact substring |\n| arch/isa_semantics_qa (instruction semantics) | 14 | 3 | rubric judge |\n| pe_import_reasoning / pe_structure_qa | 7 | 2 | rubric / exact field |\n\nSources: Ubuntu x86-64 system libraries (libcrypto, zstd, sqlite3, libxml2, zlib, libpng),\nWindows PE samples, Debian production cross-toolchain glibc (AArch64/RISC-V/MIPS/PPC),\nAndroid/Termux production aarch64 libs, and labeled cross-compiled targets.\nThe 29 agentic challenges are also packaged in Harbor task format on the HF dataset repo —\nbuilt + scored in real Docker containers: 29/29 (gold → reward 1, wrong answer → 0).\n\n## Measured difficulty (honest)\n- Agentic 29: Claude Opus 5.5 (tools: bash+read) solved 29/29 envs, 145/145 attempts, byte-exact\n  vs gold — that tier is warm-up level for a frontier agent; full table in EVAL_RESULTS-passk.md.\n- Open-weight baseline (local qwen3.8, single-shot, temp 0): 62% on our cross-arch/library slice;\n  on our separate private hard set (99 items, not in this pack) the same model solved 14/99 at pass@8 —\n  that mid-band is where RL training signal lives; we calibrate per buyer model.\n\nDatasheet + full evaluation results: https://huggingface.co/datasets/S3RL4/binary-reasoning-v1\nCanary for contamination checks: RE-Lab-brv-2026-10-29-8c1e4f7a\n\n## Usage (verifiers v1)\n```bash\nuv run vf-eval binary_reasoning_v1 --select.limit 20\n```\nOr load in code:\n```python\nfrom verifiers import load_environment\nenv = load_environment(\"binary-reasoning-v1\")   # 100-task open sample\n```\n","encoding":"utf-8","truncated":false,"total_bytes":3224},"status":null}