{"data":{"kind":"file","path":"README.md","version_id":"skugveqekbj7vcczoxy4lw1z","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":8703,"modified_at":"2026-06-03T19:11:16.638000","content_hash":"594d3eb85975c44a3278d4ff3baba5ed6c063a682d857f31bc363ddb32ef73a4"},"entries":[],"content":"# py-bug-trace\n\nPython **bug-trace** benchmark for **Laguna XS.2** — given subtly broken code, predict the *exact* printed output (not SWE-style fixes). Levels 1–2 shipped; Levels 3–5 on the roadmap.\n\n**Jump to:** [Why this benchmark](#why-this-benchmark) · [Difficulty levels](#difficulty-levels) · [Results](#results) · [Quick start](#quick-start) · [Who is this for?](#who-is-this-for) · [Choose your path](#choose-your-path) · [Task banks](#task-banks) · [Documentation](#documentation) · [Roadmap](#sprint-roadmap)\n\n## Who is this for?\n\n| Persona | Start here | What you will do |\n|---------|------------|------------------|\n| **HF / standalone clone** | [QUICKSTART.md](QUICKSTART.md) | `prime login`, hosted smoke, `./configs/sweep.yaml` |\n| **Full Prime Lab** | [lab/README.md](../../README.md) | Paths under `lab/configs/` and `environments/py_bug_trace/` |\n| **Reproduce the benchmark** | [docs/workflow.md](docs/workflow.md) | Sweep matrix, compare, optional publish |\n| **Browse artifacts only** | [Results](#results) · [docs/publish.md](docs/publish.md) | HF datasets and lab bundle (no rerun) |\n| **Contributors** | [Task banks](#task-banks) · [SPRINT_NOTES.md](docs/SPRINT_NOTES.md) | Edit task banks, sweep tooling |\n\n## Why this benchmark\n\n- **Trace, don’t fix** — models predict stdout from broken snippets; scoring is exact-match on `<answer>` (no partial credit).\n- **Harder than SWE-bench in a different way** — Python-specific gotchas (defaults, closures, async scheduling) fool strong models.\n- **Separate levels** — each level is five *different* tasks, not the same prompts with higher difficulty.\n- **Laguna vs field** — multi-model hosted sweep with local matrix compare and optional HF rollout export.\n\n## Difficulty levels\n\nEach **Coding Task Difficulty Level** is a separate set of five tasks. For shipped work, **Sprint N = Level N**.\n\n| Level | What it covers | What the model does |\n|-------|----------------|---------------------|\n| **1** | Core Python gotchas (defaults, closures, generators, …) | Predict exact output (`<answer>`) |\n| **2** | Threads and `asyncio` | Same |\n| **3–5** *(planned)* | API bugs → multi-file refactor → real codebases | Fix code; **pytest** pass/fail (not exact output) |\n\nRun and compare **one level at a time** — L1 and L2 both use example indices 0–4, but the tasks and ids differ.\n\n## Results\n\nWhat you get after running evals and compare (no fixed scores here — regenerate locally):\n\n| Output | Where |\n|--------|--------|\n| Matrix leaderboard + per-task tables | `reports/matrix/report.md`, `summary.json`, `comparison.csv`, `sweep_summary_matrix.csv` |\n| Prime eval dashboard | [Prime Evals](https://app.primeintellect.ai/dashboard/evaluations) |\n| Public rollout datasets | [poolside-laguna-hackathon/datasets](https://huggingface.co/poolside-laguna-hackathon/datasets) |\n| Full lab clone (configs + tools) | [laguna-eval-experiments](https://huggingface.co/poolside-laguna-hackathon/laguna-eval-experiments) |\n\n`reports/` is gitignored — generated on your machine after a sweep.\n\n**Regenerate** (requires completed sweep in `sweep_state.json`):\n\n```bash\n# lab root\npython environments/py_bug_trace/tools/sweep.py --compare\n# standalone: python tools/sweep.py --compare\n\n# or\npython environments/py_bug_trace/tools/compare_evals.py --from-sweep\n```\n\nPer-model HF dataset list: [docs/publish.md](docs/publish.md#published-hf-datasets-full-table).\n\nAfter a completed sweep, `python tools/sweep.py --compare` prints a cross-level leaderboard and per-task scorecards. Example (12 models, Laguna baseline):\n\n![Sweep matrix compare leaderboard](screenshots/screenshot-compare.jpg)\n\nRetroactive Hugging Face rollout export (`python tools/sweep.py --hf-publish-rollouts`) maps each level to dataset repos under `neomatrix369` and `poolside-laguna-hackathon`:\n\n![HF rollout publish summary](screenshots/screenshot-hf-push-rollups.jpg)\n\n## Quick start\n\n1. **Setup** — [QUICKSTART.md](QUICKSTART.md) (`prime login`, `uv pip install -e \".[tools]\"`).\n2. **Hosted smoke (Level 1)** — `prime eval run neomatrix369/py-bug-trace-level-1 -m poolside/laguna-xs.2 --hosted --follow`\n3. **Sweep status** — `python tools/sweep.py --status` (standalone) or `python environments/py_bug_trace/tools/sweep.py --status` (lab).\n\n   The status grid shows pass@1 %, wall time, and cost for each model×level cell (example after a full 24-cell matrix):\n\n   ![Model sweep status grid](screenshots/screenshot-sweep.jpg)\n\n4. **Full matrix** — [docs/workflow.md](docs/workflow.md) (preflight + `python tools/sweep.py`).\n\n## Choose your path\n\n| I want to… | Start here |\n|------------|------------|\n| Install and verify Prime | [QUICKSTART.md](QUICKSTART.md) |\n| Understand Prime vs env vars | [docs/setup.md](docs/setup.md) |\n| Run one hosted eval | [docs/workflow.md](docs/workflow.md#run-hosted) |\n| Run the 12-cell hosted matrix | [docs/workflow.md](docs/workflow.md#multi-model-sweep) |\n| Compare sweep results | [docs/compare.md](docs/compare.md) |\n| Fix 0/0 missing rollouts | `python tools/sweep.py --rerun-missing` · [PRIME_SUPPORT_0-0-rollouts.md](docs/PRIME_SUPPORT_0-0-rollouts.md) |\n| Publish Prime leaderboard | [docs/publish.md](docs/publish.md#prime-evals-and-leaderboard) |\n| Publish HF rollout datasets | [docs/publish.md](docs/publish.md#hugging-face-hub) |\n| Browse HF without running code | [docs/publish.md](docs/publish.md#hugging-face-hub) |\n\n## Task banks\n\nShared scoring: [`answer_extraction.py`](answer_extraction.py) (used by levels and `tools/compare_evals.py`).\n\n### Level 1 — Core Python (Sprint 1)\n\n| ID | Gotcha | Expected trap |\n|----|--------|---------------|\n| mutable-default-arg | `def f(x, lst=[])` | Model says `[2]`, correct is `[1, 2]` |\n| late-binding-closure | `lambda: i` in loop | Model says `[0,1,2]`, correct is `[2,2,2]` |\n| generator-exhaustion | `sum(gen)` twice | Model says `(12,12)`, correct is `(12,0)` |\n| wrong-base-case | `sum_digits` returns 0 for n<10 | Model says `6`, correct is `5` |\n| string-lexicographic-max | `max(['10','9','100'])` | Model says `100`, correct is `9` |\n\n`py_bug_trace_level_1.py` · package `py-bug-trace`.\n\n### Level 2 — Async/concurrency (Sprint 2)\n\n| ID | Gotcha | Expected trap |\n|----|--------|---------------|\n| thread-race-counter | `threading` race on `counter += 1` | Model says `False` or fixed count; correct is `True` |\n| timer-execution-order | `Timer(0.0)` vs synchronous `print` | Model says `timer` first; correct is `main` |\n| asyncio-gather-exception-type | `gather(..., return_exceptions=True)` | Model says `Exception`; correct is `RuntimeError` |\n| asyncio-task-cancelled | cancel + await cancelled task | Model omits handler; correct is `cancelled` |\n| asyncio-yield-before-finish | `create_task` + `sleep(0)` | Model says `False`; correct is `True` |\n\n`py_bug_trace_level_2.py` · package `py-bug-trace-level-2`.\n\n**Known issues (hosted Level 2 evals)** {#known-issues-hosted-level-2-evals}: Retired ids may appear in older exports (`asyncio-gather-order`, `asyncio-run-nested`). `timer-execution-order` may show `⚠FMT` — see [SPRINT_NOTES.md](docs/SPRINT_NOTES.md). Missing rollouts: [PRIME_SUPPORT_0-0-rollouts.md](docs/PRIME_SUPPORT_0-0-rollouts.md).\n\n## Documentation\n\nFull guides: **[docs/README.md](docs/README.md)**\n\n| Topic | Doc |\n|-------|-----|\n| Setup, Prime, env vars | [docs/setup.md](docs/setup.md) |\n| Run evals and sweep | [docs/workflow.md](docs/workflow.md) |\n| Compare reports | [docs/compare.md](docs/compare.md) |\n| Publish Prime / HF | [docs/publish.md](docs/publish.md) |\n\nMoved from this README (same content, easier navigation): prerequisites → setup; run/sweep → workflow; compare → compare; publish → publish.\n\n## Sprint roadmap\n\n| Sprint / level | Status | Scoring |\n|----------------|--------|---------|\n| 1 | shipped | Exact `<answer>` match |\n| 2 | shipped | Exact match |\n| 3–5 | planned | pytest pass/fail |\n\n## File layout\n\n| File | Role |\n|------|------|\n| `README.md` | This hub |\n| `screenshots/` | Example sweep status, compare, and HF publish output |\n| `QUICKSTART.md` | Install, login, smoke eval |\n| `.env.example` | Optional API keys |\n| `docs/` | Setup, workflow, compare, publish, sprint notes |\n| `configs/sweep.yaml` | Standalone sweep config |\n| `configs/` | Endpoints + eval TOMLs (standalone) |\n| `py_bug_trace_level_*.py` | Task banks |\n| `pkg_root.py` | Shared `PKG_ROOT` / `LAB_ROOT` paths for tools and tests |\n| `tools/` | Matrix runner (`tools/sweep.py`), compare, publish helpers |\n| `tests/` | Unit tests for tooling |\n| `sweep_state.json` / `sweep_logs/` | Progress (gitignored) |\n| `reports/` | Compare artifacts (gitignored) |\n","encoding":"utf-8","truncated":false,"total_bytes":8703},"status":null}