{"data":{"kind":"file","path":"README.md","version_id":"cwgoawp7q8q2i6g4nazdjhmi","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":7614,"modified_at":"2026-09-12T11:16:12.020000","content_hash":"33bc1694004180d11e7a4cd3db288baa85e8cb471ec9f100fdf3ae1b3a3e9796"},"entries":[],"content":"# sci-pipeline-v1\n\nClean and aggregate a messy sensor time series inside a sandbox. **The artifact is the\nanswer** — the reward reads the file the agent wrote, not what it said.\n\n| | |\n|---|---|\n| **Archetype** | Agentic, sandboxed, artifact-verified, procedural + `INFINITE` |\n| **Harness** | `bash`, `claude-code`, `codex` — anything with a shell |\n| **Runtime** | Container required (`NEEDS_CONTAINER = True`); the runtime's default image, `python:3.11-slim` |\n| **Reward** | `daily_means` (0.5) + `record_counts` (0.25) + `extremes` (0.25); binary per component — counts exact, temperatures within 0.01 °C |\n\n## The task\n\n`/workspace/readings.csv` holds hourly station readings. The agent must write\n`/workspace/result.json` with per-day mean temperatures in Celsius, valid/missing counts,\nand the extremes. An agent that describes a correct pipeline but never runs it scores zero.\n\nFour defects, each of which silently corrupts a naive aggregation:\n\n| Defect | Failure if ignored |\n|---|---|\n| `-999.0` sentinel for missing readings | Large and obvious — drags the mean to absurdity |\n| Fahrenheit rows, marked only in a `unit` column | Small and **non**-obvious — the one most likely to survive the agent's own sanity check |\n| Duplicate timestamps (first occurrence wins) | Slight mean shift, broken counts |\n| Shuffled row order | Makes \"first in file order\" differ from \"earliest by time\" — stated in the prompt, not a trick |\n\nThe offline tests assert that ignoring any single defect costs reward, so no defect is\ndecoration.\n\n## Rewards and metrics\n\nEvery reward reads the `result.json` that `finalize` captured from the sandbox. A missing\nor unparseable file scores zero on all three.\n\n| Reward | Weight | Pays 1.0 when |\n|---|---|---|\n| `daily_means` | 0.5 | The set of dates in `daily_mean_c` equals the reference exactly and every mean is within `tolerance_c` (default 0.01 °C) of the reference. A missing date, an extra date, or one mean out of tolerance scores 0. |\n| `record_counts` | 0.25 | `n_valid` and `n_missing` both equal the reference counts. Values are passed through `int()` before comparison. |\n| `extremes` | 0.25 | `max_c` and `min_c` are both within `tolerance_c` of the reference. |\n\nMetrics carry no weight; they exist so a failed rollout is diagnosable from the trace:\n\n| Metric | Meaning |\n|---|---|\n| `produced_output` | 1.0 if a parseable `result.json` existed at all — separates \"no run\" from \"wrong run\". |\n| `day_coverage` | Share of expected dates present in `daily_mean_c`, whatever their values — shows truncated output that `daily_means` scores as 0. |\n\n## Arguments\n\nTaskset knobs are passed as `--env.taskset.<name>` (kebab-case) to `eval`, or as\n`--taskset.<name>` to `validate`.\n\n| Argument | Default | Meaning |\n|---|---|---|\n| `num-days` | `5` | Calendar days of hourly readings per task. |\n| `missing-rate` | `0.06` | Share of readings replaced by the `-999.0` sentinel. |\n| `fahrenheit-rate` | `0.15` | Share of readings recorded in Fahrenheit. |\n| `duplicate-rate` | `0.05` | Share of readings re-emitted under a timestamp that already appeared. |\n| `seed` | `20260826` | Generator seed; the task stream is deterministic per seed. |\n| `task.tolerance-c` | `0.01` | Absolute tolerance, in Celsius, for the mean and extreme comparisons. |\n\n## Dependencies\n\nThe package depends only on `verifiers`; the reference pipeline is standard library\n(`csv`, `decimal`). The agent's container is the runtime's default image,\n`python:3.11-slim`: Python and the stdlib `csv` module are enough to solve the task, and no\ndata-science library is preinstalled. No secrets or environment variables are needed\nbeyond the model provider's API key.\n\n## Run\n\n```bash\nuv run validate sci-pipeline-v1 -n 10 --runtime.type docker      # model-free: keys re-derive\nuv run eval sci-pipeline-v1 -n 10 --model <model-id> \\\n  --env.agent.harness.id bash --env.agent.runtime.type docker\nuv run eval sci-pipeline-v1 -n 10 --model <model-id> \\\n  --env.agent.harness.id bash --env.agent.runtime.type docker \\\n  --env.taskset.num-days 14 --env.taskset.missing-rate 0.12     # longer series, more gaps\n```\n\nAlways pass `--env.agent.runtime.type`. Without it `eval` falls back to the Prime sandbox\nruntime, which provisions a remote, billed sandbox for every rollout.\n\n## Design notes\n\n**The reference pipeline is stdlib-only.** The agent will reach for pandas or polars; if\nthe grader used pandas too, the two would share its missing-value and rounding conventions\nand the environment would quietly stop testing whether the agent handled them. The\ncontract lives in the prompt, not in a library's defaults.\n\n**Half-up rounding, stated explicitly.** Python's `round()` is banker's rounding and would\nsend 12.345 to 12.34. The prompt promises the ordinary convention and the reference\nimplements it with `Decimal`.\n\n**Reading happens in `finalize`, not in a reward.** Rewards may run after the runtime is\nreleased. `finalize` captures the artifact while the sandbox is alive and puts both the\nsubmission and any failure reason in `trace.info`, so a failed rollout is diagnosable from\nthe trace alone.\n\n**Rewards are split three ways to be diagnostic.** Right counts with wrong means points at\naggregation; the reverse points at parsing.\n\n## Status\n\nModel-free validation: **10/10 valid** on the Docker runtime\n(`uv run validate sci-pipeline-v1 -n 10 --runtime.type docker`) — the first\ncontainer-runtime run in this repo.\n\n20 offline tests pass and ruff is clean. They check the reference against a hand-computed\nfixture (it is the answer key for every task, so an untested reference means an\nenvironment that is confidently wrong at scale), then check the rewards against the\noutputs three plausible broken pipelines actually produce and against direct attempts to\ncheat each component.\n\nThey also drive `setup` → `finalize` against a **real subprocess runtime**, covering the\nfile plumbing that scoring tests cannot reach: that `setup` materialises exactly one file,\nthat a missing artifact scores zero and records why, that a correct `result.json` scores\n1.0, and that malformed JSON is reported as malformed rather than as absent. The task is\nrelocated into a temp directory for that test because the subprocess runtime resolves\nabsolute paths against the host filesystem — it does not sandbox them, which is exactly why\nthis environment declares `NEEDS_CONTAINER` for real runs.\n\nMeasured end-to-end on 2026-09-12 via Prime Inference (temperature 0.0, `bash` harness,\nDocker runtime, 10 tasks each, every rollout completed; the reproducible command is\n`uv run eval sci-pipeline-v1 -n 10 --model <model-id> --env.agent.harness.id bash --env.agent.runtime.type docker`,\nplus `--env.taskset.num-days 30` for the longer setting):\n\n| Model | `num_days` | Reward | `daily_means` | `extremes` | `record_counts` | `produced_output` |\n|---|---|---|---|---|---|---|\n| `openai/gpt-5-mini` | 5 | **1.000** | 1.0 | 1.0 | 1.0 | 1.0 |\n| `openai/gpt-5-mini` | 30 | **1.000** | 1.0 | 1.0 | 1.0 | 1.0 |\n| `openai/gpt-5-nano` | 5 | **0.875** | 0.9 | 0.8 | 0.9 | 0.9 |\n\nReading: gpt-5-mini saturates the task at both lengths — for a competent shell agent the\ncleaning rules are fully specified and the artifact is checkable, so this environment is a\ncorrectness floor rather than a ranking eval at that capability level. gpt-5-nano does not\nsaturate it (one task produced no artifact, one got the extremes wrong), so the environment\nseparates the two. The generator's messiness is fixed; a difficulty knob on the corruption\nmix is the natural next revision if a harder ceiling is wanted.\n","encoding":"utf-8","truncated":false,"total_bytes":7614},"status":null}