{"data":{"kind":"file","path":"README.md","version_id":"g3kmu21wm8acus2rf92i22qu","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6130,"modified_at":"2026-09-13T04:06:06.956000","content_hash":"9299929dfefe556f1d411c4539e4e963a44f86646b1f291131b2c301b3a472e2"},"entries":[],"content":"# senior-swe-bench\n\nRequires `verifiers[harbor]>=0.3.1`.\n\nSnorkel AI's [Senior SWE-Bench](https://senior-swe-bench.snorkel.ai) (v2026.06) senior-engineer tasks on harbor, backed by prime's prebuilt task images. 50 public tasks (25 `investigate`, 25 `design`) over 12 OSS repos (better-auth, electric, firezone, gitea, harbor, immich, paperless-ngx, plausible, posthog, prefect, teleport, turborepo). Scored by the upstream multi-stage verifier in the task container: agent-diff capture, native pytest/vitest verifiers, LLM rubric + taste judges, and (design tasks) a validated user-stories agent; the binary \"basic\" solve lands in `/logs/verifier/reward.txt`, per-stage subscores in `reward.json`/`reward_details.json`.\n\nNeeds a **prime runtime** with access to the prebuilt task images (`senior-swe-bench/<task>:latest`, built server-side from each task's `environment/Dockerfile` — harbor tasksets don't build Dockerfiles). The LLM stages are gated behind `--env.taskset.task.upstream-judges` (default off): when on, they call **upstream's models** (`claude-sonnet-4-6` judges/validation agent, `claude-haiku-4-5` classifier) through the `--env.taskset.task.judge-client.*` endpoint — a vf `BaseClientConfig` defaulting to PI inference with the key from the Prime CLI config (load fails fast when no credential resolves); judge sub-scores are recorded as trace metrics (`verifier_score`, `rubric_*`, `taste_*`, `validation_score`), and the raw judge/verifier responses land in `trace.info` (`judge_output`, `reward_details`). Host `SSB_OVERRIDE_*` vars still override the models, and a host `PORTKEY_API_KEY` disables the judge-client wiring in favor of upstream's Portkey gateway routing. When off, the key templates are blanked so host credentials can't implicitly enable the LLM stages. Note `design` tasks score meaningfully only with `upstream_judges` — their validated user-stories agent is part of the binary reward; without it they're invalid trials upstream. Deviation from upstream: harbor excludes *invalid* trials (empty `reward.txt`, written when the verifier or validation agent itself fails) from solve rates; here they score 0.\n\n## Taskset\n\n- **Source:** [`snorkel-ai/senior-swe-bench-v2026.06`](https://github.com/snorkel-ai/senior-swe-bench-v2026.06) (pinned by commit in `SeniorSWEBenchConfig.repo`)\n- **Size:** 50 tasks\n\n## Timeouts\n\nTask-declared agent and verifier timeouts are ignored by default. Restore them with `--env.taskset.ignore-timeouts false` for a more faithful comparison to the Harbor implementation — though we advise against it, since enforcing them confounds model capability with the inference infrastructure (e.g. how many tokens/sec it serves).\n\n## Changelog\n\n- 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.\n- 2026-07-31: Custom verifier finalization now delegates to Harbor first, preserving `verifiers>=0.2.2.dev65` collect hooks and native artifacts before the multi-stage grader runs.\n- 2026-07-22: Task images resolve as org-less platform refs `senior-swe-bench/<task>:latest` (task id moves from the tag into the name; the tag becomes `latest`) instead of the team-scoped `prime/primeintellect/senior-swe-bench:<task>`. Merges in lockstep with the registry rename for the senior-swe-bench family.\n- 2026-07-15: Moved into the `swe/` group and ported to the task-centric verifiers API (`verifiers>=0.2.0`, Python `>=3.12`). The upstream verifier now runs in `finalize` (with the scoring budget mirrored onto that stage), which pulls its artifacts (`reward.txt`, `reward.json`, `judge_output.json`, `reward_details.json`) into `trace.info`; scoring reads the trace only — `solved` from `reward_txt`, the flat per-stage sub-scores as metrics whenever `reward.json` exists. Added `--env.taskset.task.upstream-judges` (default off) gating the upstream LLM stages (rubric + taste judges, design-task validation agent): on requires a judge API key on the eval host (fail-fast at load), records judge sub-scores as trace metrics and raw responses in `trace.info`, preinstalls `fastapi`/`orjson` (the verifier's floating `litellm>=1.0,<2.0` install currently lands 1.92.0, which pulls `litellm.proxy` imports on tool-call completions and dies on `ModuleNotFoundError` without them), and routes the stages through `--env.taskset.task.judge-client.*` (a vf `BaseClientConfig`, default PI inference keyed from the Prime CLI config) while keeping upstream's models — upstream's own defaults don't survive a provider swap (the direct judge slugs need `ANTHROPIC_API_KEY`; the validation agent's stock default is a Portkey gateway slug that can't route at all without a gateway key); off blanks the `[verifier.env]` key templates so host credentials can't implicitly enable the stages. Pinned `verifiers[harbor]>=0.2.1.dev40` (the base harbor taskset parses schema-1.0 size strings, verifiers#2031) and dropped the local re-parse; Python floor is `>=3.12` (`harbor` requires it). Verified: a judged rollout with no provider keys in the eval env scores rubric + taste `ok` through the default PI-inference judge client; judged 8-rollout eval records rubric/taste scores; `uv run validate --only-gold` on `immich-feat-recently-added-assets` with judges on → valid (393s).\n- 2026-07-03: Add the `validate()` oracle hook for `uv run validate` (verifiers#1905): apply the task's upstream `solution/solve.sh`, rerun the verifier, valid iff it scores 1.0 (checked on prime: noop and apply-answer on `better-auth-fix-api-key-run`). Drop the local `solved` override — `verifiers>=0.1.15.dev418` resolves `[verifier.env]` in the base harbor taskset (verifiers#1921) — and reuse the `harbor` package for env-template resolution and size-string parsing instead of local reimplementations.\n- 2026-07-02: Initial port. Pinned upstream commit `e30b0e19fdbc`; 50 prebuilt images pushed to the prime registry; `[verifier.env]` passthrough, `/repo/<REPO_NAME>` workdir, and harbor size-string resource parsing on top of the base harbor taskset. Validated: fail→0 and oracle-patch→1 on `better-auth-fix-api-key-run` via a prime sandbox.\n","encoding":"utf-8","truncated":false,"total_bytes":6130},"status":null}