{"data":{"kind":"file","path":"README.md","version_id":"rvvfg1b7ev88hai4e7adr9vh","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6054,"modified_at":"2026-08-07T06:30:03.622000","content_hash":"41adcf44fb69c29d0a143ea813e2ae2bd8cd76116e7eeebb05fd54059928e372"},"entries":[],"content":"# beautifulsoup-env\r\n\r\n### Overview\r\n- **Environment ID**: `beautifulsoup-env`\r\n- **Short description**: RL environment for HTML extraction with BeautifulSoup, graded by exact row comparison\r\n- **Tags**: beautifulsoup, html, web-scraping\r\n\r\n### Datasets\r\n- **Primary dataset(s)**: `eltociear/beautifulsoup-tasks-v1`\r\n- **Source links**: [HuggingFace Dataset](https://huggingface.co/datasets/eltociear/beautifulsoup-tasks-v1)\r\n- **Split sizes**: train (26 examples)\r\n\r\n### Task Categories\r\n\r\n| Category | Tasks | Description |\r\n|----------|-------|-------------|\r\n| selection | 7 | `find_all`, class filters, CSS selectors, attribute presence, `select_one`, href predicates |\r\n| attributes | 6 | reading and casting `data-*`, `meta` content, entity decoding, missing attributes with a default, class lists |\r\n| navigation | 5 | `find_next_sibling`, `find_parent`, direct children only (`recursive=False`), pairing a heading with its link |\r\n| tables | 4 | header cells, full row extraction with casting, filtering rows by class, aggregating a column |\r\n| malformed | 4 | unclosed `<li>`, a `<div>` inside a `<p>`, a stray `</span>`, an `<img>` with no `alt` |\r\n\r\n### ⚠ The determinism trap in this domain\r\n\r\n**BeautifulSoup's result depends on the parser** — measured, not assumed. On the malformed\r\ndocument in this dataset:\r\n\r\n```\r\nhtml.parser -> soup.find(\"p\").get_text(strip=True) == \"Intro textnested blocktail text\"\r\nlxml        -> soup.find(\"p\").get_text(strip=True) == \"Intro text\"\r\n```\r\n\r\nThey disagree about whether a `<div>` closes an open `<p>`. An answer key built with one parser\r\nand graded under another is simply wrong, so every task states\r\n`BeautifulSoup(html, \"html.parser\")` in its prompt, and `html.parser` is used because it ships\r\nwith CPython — no extra dependency can be missing or differently versioned in the sandbox.\r\n\r\n### Task\r\n- **Type**: Multi-turn tool use (default `max_turns=5`)\r\n- **Rubric overview**: Binary pass/fail on an exact row-by-row comparison\r\n\r\n### Quickstart\r\n\r\n```bash\r\nuv run vf-eval beautifulsoup-env -p prime -m openai/gpt-5.4-nano -s\r\n```\r\n\r\n### Tools Available\r\n- `execute_code(code: str)` — `html` and `soup` (already parsed with `html.parser`) are in\r\n  scope, along with `BeautifulSoup` and `re`; `result` persists across turns\r\n- `bash(command: str)` — run shell commands in the sandbox\r\n\r\nThe model must leave `result` as a **list of lists of plain values**, never `Tag` or\r\n`NavigableString` objects.\r\n\r\n### Grading\r\n\r\nRow order, row count and values are all compared exactly. There is **no float tolerance**:\r\neverything here is text or a count, and several tasks say \"as an int\", so returning the string\r\n`\"14\"` where an int was asked for fails — casting the extracted text is part of the work.\r\n\r\nThe comparator has its own unit checks rather than being trusted, including that correct rows in\r\nthe **wrong order fail**, that `\"14\"` does not pass for `14`, and that `True` does not pass\r\nfor `1`.\r\n\r\nA wrong answer scores `0.0` silently. A broken scorer scores `0.0` and additionally sets\r\n`state[\"scoring_error\"]`, so eval runs can tell \"the model was wrong\" from \"the harness failed\".\r\n\r\n### Building the dataset\r\n\r\n```bash\r\npython build_tasks.py --verify          # check every task\r\npython build_tasks.py --out train.jsonl # regenerate\r\n```\r\n\r\n`--verify` checks each task for clean execution, determinism across two fresh parses, a\r\nnon-empty list of **JSON-safe primitives** — which is what catches the common error of returning\r\na `Tag` instead of extracted text — no accidental empty strings, and an exact serialisation\r\nround-trip. All 26 pass on beautifulsoup4 4.15.0.\r\n\r\n## Environment arguments\r\n\r\n`load_environment()` deliberately exposes very little: the program's guidance is that an\r\nenvironment should have one correct way to be run, so nothing about the prompts, the parsing or\r\nthe grading is configurable.\r\n\r\n| Argument | Default | Meaning |\r\n|---|---|---|\r\n| `split` | `'train'` | Dataset split to load. |\r\n| `dataset_name` | `'eltociear/beautifulsoup-tasks-v1'` | Hugging Face dataset of tasks. Change only to point at a fork. |\r\n| `max_turns` | `5` | Tool-use turns the model gets before the rollout ends. |\r\n| `**kwargs` | — | Passed through to the underlying `SandboxEnv`. |\r\n\r\n## Reward rubric\r\n\r\n| Reward function | Weight | What it returns |\r\n|---|---|---|\r\n| `correctness` | 1.0 | Binary: 1.0 when the answer matches the reference, else 0.0. |\r\n\r\nThere is no LLM judge and no partial credit. The score is computed on the host in\r\n`post_rollout` and read back by the rubric, so the reward is a deterministic function of the\r\nvalues the model left in `result`. A **harness** failure (dead sandbox, unreadable read-back)\r\nalso scores 0.0 but additionally sets `state[\"scoring_error\"]`, so an eval run can tell a\r\nbroken harness from a wrong answer instead of blaming the model.\r\n\r\n## Dependencies\r\n\r\n`beautifulsoup4>=4.12.0`, `datasets>=4.1.0`, `verifiers>=0.1.8`\r\n\r\n## Sample `vf-eval` usage\r\n\r\n```bash\r\nuv run vf-install beautifulsoup-env\r\nuv run vf-eval -s beautifulsoup-env -m gpt-4.1 -n 5 -r 3\r\nuv run vf-tui                      # inspect the outputs/ folder it writes\r\n```\r\n\r\n### Known limitation\r\n\r\nThe Docker **sandbox transport** has not been executed — `docker run`, `pip install`, and the\r\nmodel's code running inside the container — because that needs a Docker runtime and an\r\ninference provider key, neither available where this was authored.\r\n\r\nEverything else is exercised. `environments/test_scoring_path.py` runs this environment's\r\n`post_rollout` and `Rubric` with the sandbox mocked out, asserting that a correct answer scores\r\n1.0, a well-formed wrong answer scores 0.0, and a dead sandbox scores 0.0 *and* sets\r\n`scoring_error` so a harness failure is never mistaken for a bad model. It also checks that\r\nwhat `build_tasks.py` emits is exactly what the scorer expects. The environment mirrors the\r\nstructure of `polars_env` (already accepted into the Environments Program) and imports cleanly\r\nagainst `verifiers` 0.2.1.\r\n","encoding":"utf-8","truncated":false,"total_bytes":6054},"status":null}