{"data":{"kind":"file","path":"README.md","version_id":"g8rw9wempreiykvmor93lcq8","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":7076,"modified_at":"2026-10-30T01:45:52.650000","content_hash":"df1cd25cb97170aa6420952bd32f8dc07aec9fdc07ae8d8d8a6209779b48b74c"},"entries":[],"content":"# transitbench\n\nTransitBench is a benchmark for autonomous scientific programming in astronomy. A coding agent gets one year of real Kepler light curves for 196 labelled dev stars and a weak starter search, and builds a planet-search pipeline in Python. When the session ends, the pipeline is frozen and runs once, with no language model, on 2,000 hidden stars. There it has to find the planets I injected and stay quiet on eclipsing binaries and on stars where Kepler's own search fired on something that wasn't a planet. It's scored on planets found minus false candidates. The code is also on GitHub at [eigenpaul/transitbench](https://github.com/eigenpaul/transitbench).\n\n## Tasksets\n\n- **Source:** Kepler DR25 long-cadence light curves for quarters 10 to 13. The dev stars ship in the survey image, with their answers in [dev/truth.json](dev/truth.json).\n- **Size:** `transitbench-survey` is the benchmark. Its task `survey-r1` grades a submission on the hidden samples A and B, 1,000 stars each, and on the dev set in one pass, and `survey-dev` grades it on the dev stars only. `transitbench-mini` is for RL training. Its train split never ends and its dev split has 32 tasks. Each task puts 12 labelled stars in the box and grades the solution on 16 hidden stars. Half of those have an injected planet.\n- **Reward:** S, which is planets found minus false candidates over the number of planets (see Grading), on A for `survey-r1` and on dev for `survey-dev`. S, completeness C, false candidates per 100 scored stars and the share of flagged binaries called a planet are metrics for every graded set (`S_B`, `C_A`, `F_per_100_A`, `eb_as_planet_A` and so on). Mini's reward is clip((credit - false candidates) / 8, -1, 1) on its hidden stars.\n\nA survey session runs for up to four hours in a box with 16 CPUs and 32 GiB of memory. Its TASK.md allows 60 turns and 200,000 tokens. A mini session runs in a smaller box for up to an hour, and its TASK.md allows 12 turns and 32,768 tokens. The box has no network, and each bash command is killed after 940 s (75 s in mini). Grading runs in a second, fresh box. Outside the evaluator only `survey-dev` runs, because A and B, the mini pool and the secret that draws mini tasks stay private.\n\n## Running\n\n`prime env install` puts the package and verifiers into the active virtualenv. Pin verifiers to 0.4.1.dev84, the version the results used, since the floor alone takes the newest release. The commands run in Prime sandboxes from the images pinned in [images.json](transitbench_env/data/images.json), so they need `prime login`, and the sandbox time is charged to your Prime account. `TRANSITBENCH_DATA` can be any writable directory, and each run keeps its solution archive and per-star grade there. The model-free check grades a gold solution, which has to score exactly 1, and the untouched starter, which has to score below 1:\n\n```\nuv venv --python 3.12 && source .venv/bin/activate\nprime env install eigenpaul/transitbench@1.0.0\nuv pip install \"verifiers==0.4.1.dev84\"\nexport TRANSITBENCH_DATA=$PWD/tb-data && mkdir -p $TRANSITBENCH_DATA\nvf-validate transitbench-survey --select.include.ids '[\"survey-dev\"]'\n```\n\nTo run a model, keep the `survey-dev` selection too. `survey-r1` needs the private samples, so vf-eval would spend a whole session on it and then fail grading with TB-E001. TASK.md tells the agent it has 60 turns and 200,000 tokens, but only `--env.agent.max-turns` and `--env.agent.max-total-tokens` enforce them, so the command below sets the caps and timeouts the results used. `-m` takes a Prime Inference model id and defaults to `deepseek/deepseek-v4-flash`, and `--client.base-url` with `--client.api-key-var` points vf-eval at another OpenAI-compatible endpoint. Without `--no-push` the run is uploaded to your Prime account.\n\n```\nvf-eval transitbench-survey -m <model> --select.include.ids '[\"survey-dev\"]' --env.agent.max-turns 60 \\\n  --env.agent.max-total-tokens 200000 --env.agent.runtime.idle-timeout 14400 --sampling.max-tokens 32768 --no-push\n```\n\nThe smoke test runs on the source, with no evaluator data, no network after the install and no model. [scripts/smoke_test.py](scripts/smoke_test.py) checks the scorer on the public dev truth, runs the starter through the grading runner on three synthetic stars (an injected planet, a 6% deep eclipse and pure noise), and checks the baseline table below against the release's digests. It ends with `smoke test: 3 of 3 checks passed`.\n\n```\nprime env pull eigenpaul/transitbench@1.0.0 -t transitbench && cd transitbench\nuv run --python 3.12 --group build python scripts/smoke_test.py   # macOS needs --python 3.13 for os.waitid\n```\n\n## Grading\n\nThe agent gets one prompt, TASK.md, rendered from [this template](transitbench_env/templates/TASK.md), and a shell in a box with the labelled stars and their answers, the starter in `solution/`, and `run_dev.py`, which scores the current solution on the labelled stars the way grading does. Grading runs `solution/solution.py` once per star in a fresh process with 60 s (20 s in mini), one core and no network, and keeps the 3 highest-scoring `period_days,t0_days,score` rows it prints. A row within 1% of a planet's period, with at least 75% of the transits lining up, earns full credit ([match.py](transitbench/match.py)). At a third, half, double or triple the period it earns half, if at least 75% of the transits line up counted from one side and 20% from the other. Every row left over is a false candidate, and S = (credit - false candidates) / planets ([score.py](transitbench/score.py)), so reporting nothing scores 0 and a perfect pipeline 1. The full rules are in the [GitHub README](https://github.com/eigenpaul/transitbench#the-task-and-the-score).\n\n## Results\n\n[results/baselines.json](results/baselines.json) holds five reference pipelines that use no language model:\n\n| Row | What it is | S on A | S on B | S on dev |\n| --- | --- | ---: | ---: | ---: |\n| `empty` | no rows on any star | 0 | 0 | 0 |\n| `starter` | the BLS search the agent starts from ([solution.py](workspace/solution/solution.py)) | 0.046 | 0.076 | 0.213 |\n| `naive_v0` | a plain BLS search with no vetting ([solution.py](baselines/naive_v0/solution.py)) | -0.159 | -0.155 | -0.038 |\n| `learned_vetter` | up to three BLS peaks per star, kept or dropped by a gradient-boosted classifier trained on labelled stars outside dev, A and B ([features.py](baselines/learned_vetter/features.py)) | 0.696 | 0.673 | 0.700 |\n| `reference_detector` | reads the answers and claims every planet with MES >= 7.1 at its true ephemeris | 0.805 | 0.805 | 0.813 |\n\nThe reference detector cheats on purpose. It scores what a perfect search at Kepler's detection threshold would, and a pipeline that finds planets below MES 7.1 can beat it.\n\nI ran nine models, five `survey-r1` submissions each. GPT-6 Astra has the highest mean S on A, 0.747, though its lead over GPT-6.1 Sol and Claude Opus 5.5 is within the noise. Every model's scores, traces and frozen solutions are at [transitbench.eigenpaul.com](https://transitbench.eigenpaul.com/).\n","encoding":"utf-8","truncated":false,"total_bytes":7076},"status":null}