{"data":{"kind":"file","path":"README.md","version_id":"vetwhuygkpza2scnkatpyf87","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5846,"modified_at":"2026-08-07T06:30:03.632000","content_hash":"61d89375519ea8b6b16eaa5af30f6f7175e61af58acf5f036ae79a580ab21833"},"entries":[],"content":"# verifiers-env\r\n\r\n### Overview\r\n- **Environment ID**: `verifiers-env`\r\n- **Short description**: RL environment for the `verifiers` library itself — parsers, reward functions and Rubric composition\r\n- **Tags**: verifiers, reward-modeling\r\n\r\n### Datasets\r\n- **Primary dataset(s)**: `eltociear/verifiers-tasks-v1`\r\n- **Source links**: [HuggingFace Dataset](https://huggingface.co/datasets/eltociear/verifiers-tasks-v1)\r\n- **Split sizes**: train (22 examples)\r\n\r\n### Why this environment\r\n\r\n`verifiers` is the library every environment in this program is written against, so the skill\r\nit tests is the one a contributor actually needs: parsing a model's output, writing reward\r\nfunctions, and composing them into a Rubric whose weighting does what you meant.\r\n\r\n| Category | Tasks | Description |\r\n|----------|-------|-------------|\r\n| parsers | 7 | `Parser`, `XMLParser` fields and `answer_field`, `ThinkParser`, custom `extract_fn`, a missing tag |\r\n| rubrics | 7 | weighted rewards, per-function metrics, a single unweighted func, a zero-weighted func still recording its metric, reward funcs reading `state[\"answer\"]` |\r\n| integration | 5 | parser + rubric together, correct vs incorrect answers, correctness blended with a format reward, `ThinkParser` end to end, determinism of repeated scoring |\r\n| format_rewards | 3 | `get_format_reward_func()` on well-formed vs bare completions, and that well-formed scores strictly higher |\r\n\r\n### Why it is gradeable with no model in the loop\r\n\r\nMeasured before any task was written: `Rubric.score_rollout(state)` is a coroutine returning\r\n`None` that mutates `state` **in place**, setting `state[\"reward\"]` and `state[\"metrics\"]`. With\r\nplain Python reward functions there is no sampling anywhere:\r\n\r\n```python\r\nRubric(funcs=[f1, f2], weights=[0.3, 0.7])\r\n# -> state[\"reward\"]  == 0.65\r\n# -> state[\"metrics\"] == {\"f1\": 1.0, \"f2\": 0.5}\r\n```\r\n\r\nParsers are likewise pure string transforms. So a library about scraping signal out of sampled\r\nmodel output is itself graded with no sampling at all.\r\n\r\n### ⚠ One thing deliberately avoided\r\n\r\n**Nothing grades a private attribute.** `Rubric` exposes `_get_reward_weights` and similar\r\nunderscore-prefixed internals but no public `reward_funcs`; an answer key built on those would\r\nbreak on any refactor of the very library this environment exists to teach. Tasks use only the\r\ndocumented surface plus the `state` keys that scoring is defined to write.\r\n\r\n### Task\r\n- **Type**: Multi-turn tool use (default `max_turns=5`)\r\n- **Rubric overview**: Binary pass/fail on a row comparison — exact on order, arity and type; floats to a narrow tolerance\r\n\r\n### Quickstart\r\n\r\n```bash\r\nuv run vf-eval verifiers-env -p prime -m openai/gpt-5.4-nano -s\r\n```\r\n\r\n### Tools Available\r\n- `execute_code(code: str)` — `vf`, `asyncio` and `json` are in scope; `result` persists across turns\r\n- `bash(command: str)` — run shell commands in the sandbox\r\n\r\n### Grading\r\n\r\nOrder, arity and the string/number distinction are exact. Floats carry a narrow relative\r\ntolerance because weighted rewards are floating-point sums — a rubric weighted `[0.3, 0.7]` over\r\nscores `1.0` and `0.5` evaluates to `0.6499999999999999`, and failing a correct rubric over that\r\nlast bit would be grading IEEE-754 rather than the library.\r\n\r\n### Building the dataset\r\n\r\n```bash\r\npython build_tasks.py --verify          # check every task\r\npython build_tasks.py --out train.jsonl # regenerate\r\n```\r\n\r\nAll 22 pass on verifiers 0.2.1.\r\n\r\n## Environment arguments\r\n\r\n`load_environment()` deliberately exposes very little: the program's guidance is that an\r\nenvironment should have one correct way to be run, so nothing about the prompts, the parsing or\r\nthe grading is configurable.\r\n\r\n| Argument | Default | Meaning |\r\n|---|---|---|\r\n| `split` | `'train'` | Dataset split to load. |\r\n| `dataset_name` | `'eltociear/verifiers-tasks-v1'` | Hugging Face dataset of tasks. Change only to point at a fork. |\r\n| `max_turns` | `5` | Tool-use turns the model gets before the rollout ends. |\r\n| `**kwargs` | — | Passed through to the underlying `SandboxEnv`. |\r\n\r\n## Reward rubric\r\n\r\n| Reward function | Weight | What it returns |\r\n|---|---|---|\r\n| `correctness` | 1.0 | Binary: 1.0 when the answer matches the reference, else 0.0. |\r\n\r\nThere is no LLM judge and no partial credit. The score is computed on the host in\r\n`post_rollout` and read back by the rubric, so the reward is a deterministic function of the\r\nvalues the model left in `result`. A **harness** failure (dead sandbox, unreadable read-back)\r\nalso scores 0.0 but additionally sets `state[\"scoring_error\"]`, so an eval run can tell a\r\nbroken harness from a wrong answer instead of blaming the model.\r\n\r\n## Dependencies\r\n\r\n`datasets>=4.1.0`, `verifiers>=0.2.0`\r\n\r\n## Sample `vf-eval` usage\r\n\r\n```bash\r\nuv run vf-install verifiers-env\r\nuv run vf-eval -s verifiers-env -m gpt-4.1 -n 5 -r 3\r\nuv run vf-tui                      # inspect the outputs/ folder it writes\r\n```\r\n\r\n### Known limitation\r\n\r\nThe Docker **sandbox transport** has not been executed — `docker run`, `pip install`, and the\r\nmodel's code running inside the container — because that needs a Docker runtime and an\r\ninference provider key, neither available where this was authored.\r\n\r\nEverything else is exercised. `environments/test_scoring_path.py` runs this environment's\r\n`post_rollout` and `Rubric` with the sandbox mocked out, asserting that a correct answer scores\r\n1.0, a well-formed wrong answer scores 0.0, and a dead sandbox scores 0.0 *and* sets\r\n`scoring_error` so a harness failure is never mistaken for a bad model. It also checks that\r\nwhat `build_tasks.py` emits is exactly what the scorer expects. The environment mirrors the\r\nstructure of `polars_env` (already accepted into the Environments Program) and imports cleanly\r\nagainst `verifiers` 0.2.1.\r\n","encoding":"utf-8","truncated":false,"total_bytes":5846},"status":null}