{"data":{"kind":"file","path":"README.md","version_id":"turubemkv9t5ya7wo7hc24yt","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":4408,"modified_at":"2026-08-01T14:18:04.707000","content_hash":"42724d6020c179192085746d28cc8f4f1873de3ee42d72c16b6200af1c1f6559"},"entries":[],"content":"# slopcodebench-v1\n\n[SlopCodeBench](https://www.scbench.ai) ([arXiv:2603.24755](https://arxiv.org/abs/2603.24755)) as a\n[verifiers](https://github.com/PrimeIntellect-ai/verifiers) v1 environment, built from the published\n[Harbor dataset](https://hub.harborframework.com/datasets/gabeorlanski/slopcodebench/latest)\n(36 problems, 196 checkpoints).\n\n## Two modes\n\n**`sequential` (default — the faithful SCB protocol).** One episode per problem: the\nexported `SlopcodebenchSequentialEnv` provisions one runtime and plays the checkpoints\nin order — a *fresh* harness session per checkpoint over a persistent `/app`, exactly\nlike SCB's native runner. Each checkpoint's own verifier (pytest incl. regression tests\n+ `scb-check`) scores the state the agent left behind, and because every step runs in\nthe same box, the erosion/verbosity *transition* metrics are meaningful. SCB's 2h\nper-checkpoint wall clock is enforced (`--env.taskset.enforce-agent-timeout false` to lift).\n\n**`flat`.** Every (problem, checkpoint) pair becomes an independent task seeded from the\n*previous checkpoint's reference solution* — stationary and dense, the right shape for\nRL rollouts and cheap ablations, but deliberately not the compounding-self-degradation\nprotocol. Pair it with the default single-agent env:\n`--env.id single-agent --env.taskset.mode flat`.\n\n**Signals** (per checkpoint trace): reward `strict_pass_rate`; metrics `solved`\n(strict == 1.0, the leaderboard's checkpoint solve), `core_pass_rate`,\n`isolated_pass_rate`, `verbosity`, `erosion`, `verbosity_increased`, `erosion_increased`\n(the last two only meaningful in sequential mode).\n\n## OpenCode harness\n\nThe package ships an OpenCode harness (`--env.agent.harness.id slopcodebench-v1`)\nmirroring SCB's leaderboard invocation: `opencode run --format=json --thinking\n--dangerously-skip-permissions --variant=high`, pinned to OpenCode 1.0.134, model calls\nrouted through the interception endpoint. Knobs: `--env.agent.harness.version`,\n`--env.agent.harness.thinking low|medium|high|xhigh|None`.\n\n## Setup\n\n```bash\nuv pip install -e .           # inside your verifiers checkout / project\n./scripts/build-images.sh     # build the per-task Docker images locally\n```\n\nThe dataset downloads to `~/.cache/harbor/` via the Harbor CLI on first load. The\ndataset ships Dockerfiles, which verifiers doesn't build — build them locally with the\nscript (tags match the default `image_prefix`), or push with `prime images push` and\nset `--env.taskset.image-prefix` to your registry prefix.\n\n## Run an eval\n\n```bash\n# faithful protocol, one problem, first 2 checkpoints (smoke test)\nuv run eval slopcodebench-v1 \\\n  -m deepseek/deepseek-v4-flash \\\n  --env.agent.harness.id slopcodebench-v1 \\\n  --env.taskset.problems '[\"xjq\"]' --env.taskset.max-checkpoint 2\n\n# 5 problems, up to 6 checkpoints each\nuv run eval slopcodebench-v1 \\\n  -m deepseek/deepseek-v4-flash \\\n  --env.agent.harness.id slopcodebench-v1 \\\n  --env.taskset.problems '[\"code_search\",\"xjq\",\"file_backup\",\"etl_pipeline\",\"datagate\"]' \\\n  --env.taskset.max-checkpoint 6\n\n# everything (36 episodes / 196 checkpoints)\nuv run eval slopcodebench-v1 -m <provider/model> --env.agent.harness.id slopcodebench-v1\n\n# flat mode (RL-shaped): independent checkpoints from reference seeds\nuv run eval slopcodebench-v1 -m <provider/model> \\\n  --env.id single-agent --env.taskset.mode flat\n```\n\nTaskset knobs (`--env.taskset.*`): `mode`, `problems` (`LITE_PROBLEMS` in `taskset.py`\nis SCB's official 5-problem lite split), `max-checkpoint`, `enforce-agent-timeout`,\n`include-prior-specs` (flat only), `image-prefix`, `dataset`.\n\n## Oracle validation\n\n`Task.validate` seeds each step's *own* reference solution and asserts the verifier\nscores it `strict_pass_rate == 1.0` — run it to confirm images/verifier plumbing\nbefore spending model tokens:\n\n```bash\nuv run validate slopcodebench-v1 --only-gold --runtime.type docker \\\n  --taskset.mode flat --taskset.problems '[\"xjq\"]'\n```\n\n(Use flat mode for validation: it enumerates every checkpoint as its own task.)\n\n## Pool harness (Laguna-native)\n\nFor Poolside Laguna models, `--env.agent.harness.id scb_pool` runs Poolside's own\n`pool` agent in automated mode (`pool exec`) — the harness Poolside uses internally\nfor RL and evals. Laguna's custom tool-call format and interleaved thinking degrade\nin generic OpenAI-style harnesses; use `scb_pool` for faithful Laguna numbers.\n","encoding":"utf-8","truncated":false,"total_bytes":4408},"status":null}