{"data":{"kind":"file","path":"README.md","version_id":"etk2chbtqjl7weudmr8lqouy","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3354,"modified_at":"2026-10-24T20:01:30.992000","content_hash":"e061346c27c8867814bb07f0b35f75f99a1df7bdb9d64df2efe0514b2c90c229"},"entries":[],"content":"# Shoot Scheduling\n\nAn environment where a model plays line producer. It receives a realistic film\nshoot and must return a schedule that respects every constraint. The grader is\nexact: the plan is valid or it is not.\n\n## Why this domain\n\nScheduling is a constraint-satisfaction problem with a mechanically checkable\nanswer and no unique solution. That makes it resistant to memorisation (a model\ncannot recall the answer) and impossible to game (the grader checks each\nconstraint independently).\n\nReal production work: scenes, sets, actors, days, availability, precedence.\n\n## The task\n\nThe model gets:\n\n- a number of shooting days\n- a list of scenes, each bound to a set and a cast\n- actor unavailability windows\n- precedence constraints between scenes\n\nIt must return one JSON object mapping each scene to a day:\n\n```json\n{\"S1\": \"D2\", \"S2\": \"D4\", \"S3\": \"D1\", \"S4\": \"D3\"}\n```\n\nA plan is valid only if all five constraints hold:\n\n1. every scene is scheduled exactly once,\n2. no two scenes share a set on the same day,\n3. no two scenes share an actor on the same day,\n4. no actor is scheduled on a day they are unavailable,\n5. every precedence constraint is respected.\n\n## Reward\n\nBinary. `1.0` if the plan is valid, `0.0` otherwise. No partial credit, no LLM\njudge, no rubric subjectivity.\n\n## Difficulty\n\nThree levels are interleaved in the dataset. Measured on Gemini 3.6 Flash:\n\n| Configuration | Pass rate |\n|---|---|\n| 3 scenes / 3 days / 3 sets | 100 % |\n| 4 scenes / 4 days / 4 sets | 50 % |\n| 4 scenes / 6 days / 3 sets | 0 % |\n\nThe measured aggregate is **33 %** over a 9-task sample.\n\nAn environment that a model passes at 100 % gives no training signal, and one it\nfails at 0 % gives none either. This one is calibrated in the useful band.\n\n## Generation\n\nThe generator draws a **valid plan first**, then derives sets and casts from it.\nThe solution therefore exists by construction and no instance is discarded at\nruntime. The cast for each day is a disjoint partition of the actor pool, which\nmakes the actor constraint satisfiable without search.\n\n## Running\n\n```bash\nuv venv vfenv1\nVIRTUAL_ENV=$PWD/vfenv1 uv pip install \"verifiers>=0.4\" datasets\nVIRTUAL_ENV=$PWD/vfenv1 uv pip install -e .\n\nvfenv1/bin/vf-eval shoot_scheduling -m gemini-3.6-flash \\\n  --client.base-url https://generativelanguage.googleapis.com/v1beta/openai \\\n  --client.api-key-var GOOGLE_API_KEY \\\n  -r 1 -c 2 --select.limit 9 --no-push \\\n  --env.agent.runtime.type subprocess\n```\n\n`--env.agent.runtime.type subprocess` is required: without it, a task with no\ntools tries to provision the cloud sandbox and dies on `SandboxError`.\n\n## Configuration\n\n| Field | Default | Meaning |\n|---|---|---|\n| `num_tasks` | 200 | number of tasks generated |\n| `seed` | 20261024 | seed of the generator, makes the dataset reproducible |\n\n## Known limitations\n\n- The instance is fully in the prompt, so a long enough context always contains\n  the solution. The difficulty comes from the constraint search, not from\n  hidden information.\n- Very large instances (more than 6 scenes on 3 sets) are usually unsatisfiable.\n  The generator avoids them by construction.\n- Some upstream provider calls fail intermittently on free endpoints. That is\n  transport, not environment.\n\n## Files\n\n- `shoot_scheduling/taskset.py` — generator, grader, taskset\n- `pyproject.toml` — package metadata\n","encoding":"utf-8","truncated":false,"total_bytes":3354},"status":null}