{"data":{"kind":"file","path":"README.md","version_id":"asq81mgiacthwgl4rbr6azrb","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":14061,"modified_at":"2026-08-28T10:24:28.594000","content_hash":"2ec0bc8ec60ff6262cf1d2dd02ffdb4501868d48989c451c02635a693ea90172"},"entries":[],"content":"# html-extraction\n\nPull seven structured fields out of synthetic product pages carrying the markup defects\nthat make real scraping hard: decoy prices, mis-encoded text, relative URLs that resolve\nagainst a `` you have to notice, lazy-loaded images, hidden duplicate product\nrecords, recommendation carousels, and per-option variants where one option is the\nselected one.\n\nSingle-turn. Deterministic. **No sandbox, no container, no code execution.** The model\nemits JSON; the grader compares JSON to an answer key generated from the template\nparameters. A 120-row run takes a few minutes and costs $0.14–0.50 depending on the model.\n\n## The task\n\nThe model gets one product page and returns one JSON object:\n\n```\ntitle - the product's name, as a human would read it on the page\nprice_cents - the CURRENT price in minor units, as an integer\ncurrency - ISO 4217 code\nin_stock - true or false\nsku - the product's stock-keeping unit\nimage_urls - absolute URLs of the PRODUCT GALLERY images only, in document order,\n deduplicated, excluding site furniture and recommended products, and\n scoped to the selected option where the gallery is tagged by option\ndispatch_date - \"YYYY-MM-DD\", from the stated today plus the stated lead time\n```\n\nReward is **binary over all seven fields**. Partial credit would pay for getting the easy\nfields and guessing the rest.\n\n## The nineteen pathologies\n\nEvery fixture draws a subset. Counts are over the published 120 rows.\n\n| pathology | n | what it does |\n|---|---|---|\n| `variant_selection` | 83 | several options, each with its own price, SKU, stock and lead time; the one that counts is marked selected, and is never distinguishable by position, price, stock or lead |\n| `variant_gallery` | 76 | every option's images ship in one gallery, **interleaved**, tagged by option |\n| `mojibake_title` | 33 | UTF-8 shown as cp1252 — the classic `’` |\n| `breadcrumb_title_decoy` | 31 | breadcrumb text that reads like a product title |\n| `hidden_decoy` | 30 | a complete, plausible, wrong product in a hidden block, before or after the real one |\n| `delivery_decoy` | 30 | a delivery date on the page that is not the dispatch date |\n| `stock_via_class` | 30 | availability is an opaque CSS class token; the page ships the stylesheet that renders it as text |\n| `decoy_old_price` | 30 | a struck-through \"was\" price |\n| `ldjson_contradiction` | 29 | an `ld+json` block whose values are drawn **independently** of the truth |\n| `working_day_dispatch` | 29 | the lead time is in working days |\n| `sku_in_attribute` | 27 | SKU only in a `data-` attribute |\n| `thousands_separator` | 26 | `£1,299.00` |\n| `relative_images` | 25 | relative, protocol-relative and absolute URLs mixed, with a `` that is **not** the quoted page URL |\n| `lazy_loaded_images` | 25 | real URL in `data-src`; `src` is a placeholder |\n| `inline_brand_mark` | 23 | the gallery ships as two containers with a brand mark between them, in neither |\n| `euro_number_format` | 20 | `1.299,00 €` — every separator means the opposite |\n| `member_price_decoy` | 20 | a cheaper members-only price |\n| `ambiguous_currency` | 16 | `$` plus a visible statement of which dollar it is |\n| `duplicate_gallery` | 8 | an image repeated, sometimes in a different URL form, so de-duplication happens after resolution |\n\n`malformed_markup` and `split_currency` were **removed** after being measured inert — a\ntolerant parser returns an identical element sequence for the first, and `get_text`\nreassembles the second into the string a human reads.\n\n## Every pathology is load-bearing\n\nHonest scraper with exactly one defence removed, scored on the rows carrying that trap:\n\n| defence removed | score on affected rows |\n|---|---|\n| mojibake repair | **0.000** |\n| the CSS stock rule | **0.000** |\n| `` resolution | **0.000** |\n| `data-src` | **0.000** |\n| excluding the inline brand mark | **0.000** |\n| de-duplication | **0.000** |\n| the working-day rule | 0.121 |\n| reading which option is selected | 0.201 |\n| ignoring hidden content | 0.531 |\n\nDistractors never coincide with the answer: ld+json name/sku/price 0/655 each, breadcrumb\n0/642, hidden title 0/719, struck price 0/609, members price 0/613, recommended-product\ntitle and SKU 0/3000.\n\n## The answer is always determinable\n\nA scraper using only semantic HTML — `hidden`/`aria-hidden`/`display:none`,\n`aria-selected`/`checked`, the CSS `content:` rules, ``, `data-src`, ``,\ncontainer membership — and visible text scores **1.000 on all seven fields**, over the\npublished 120, 840 fresh rows and the held-out set.\n\nThis is checked automatically (`scripts/leak_scan.py`) because it is the failure mode that\nmatters most and the one this environment got wrong twice: a fix that closes a shortcut by\nputting something on the page that nothing distinguishes makes the field *unanswerable*,\nand that is worse than any leak. Both times it cost a paid model run to notice.\n\n## Why there is nothing to hack in the grader\n\nThis environment exists because a sibling environment produced five reward hacks, every one\nrooted in agent-controlled code running where the verdict was computed. So:\n\n> **No agent code executes anywhere.** The model emits data; the grader compares data.\n\nThirteen adversarial passes have tested that and it has held every time:\n\n- **Every trivial baseline scores exactly 0.000** — empty, prose, echoing the page, `{}`,\n all-null, and an *oracle* per-field majority class computed from the answer key.\n- **Format alone earns nothing.** Extra or missing key → 0. Every wrong type → 0.\n Reordered `image_urls` → 0. Shotgunning fails: answer extraction takes the last\n **top-level** object, so a model is graded on its commitment.\n- **Malformed output fails closed and never crashes.** 100,000-deep nesting before the\n answer still scores 1.0 in 0.02 s; 500,000 stray braces score 0.0 in 0.02 s.\n `RecursionError` is caught explicitly.\n- **The grader cannot be reached.** Single turn, no shell, no tools, no network, no judge,\n no `eval`/`exec`, no shared state. Nothing the model emits touches the answer key.\n- The answer key comes from the template parameters, never from parsing the rendered HTML,\n so the grader and the key cannot share a parsing bug.\n\n## Accepted residuals\n\nPer this project's red-team protocol, every exploit that is accepted rather than fixed gets\na line here. These are all **partial** — each is strictly worse than doing the work, and\nnone can carry a row on its own, because reward is a conjunction over seven fields.\n\n- **The dispatch date falls to a per-lead lookup table — 0.512 against 0.115 for not doing\n the arithmetic.** Lead times are drawn from a closed ten-value set, and the modal\n calendar-offset per lead is learnable. This is the residue of fixing a *constant*-offset\n shortcut: the per-lead ceiling of ~0.55 is inherent to calendar arithmetic when the start\n weekday is uniform. Closing it properly needs per-page holidays, which is a task change.\n- **\"The longest contiguous run of `![]()
`\" substitutes for knowing what site furniture is\n — 0.493 against 0.000** on rows without `variant_gallery`. Recommendation cards hold 1–5\n images and the gallery 2–4, so the gallery is the strictly longest run only 24% of the\n time, but ties leave about half.\n- **On `variant_gallery` rows, filtering every image by `option id == selected` scores\n 0.849**, because furniture carries a matching id only ~17% of the time. ⚠️ The tempting\n fix — biasing furniture ids toward the *selected* option — **must not be applied**: that\n is a leak this project already shipped and removed, where the bias turned the\n recommendation rail into a majority-vote pointer at the answer (0.62 against a 0.635\n ceiling). Raise it only by adding more furniture ids drawn **uniformly**.\n- **Two cosmetic fingerprints.** The hidden block is the only element whose `style`\n attribute contains a `;` (719/719, no false positives), and the mojibake alphabet is a\n 13-entry table. Neither is a capability skip: both select exactly what an honest reader\n selects, by a route no simpler than the honest one.\n- **A composite lazy scraper** using the first three of these scores **0.636** against\n **1.000** honest and **0.000** for doing none of the work.\n- **Two fail-closed fragilities.** An unbalanced `{`, or an odd number of `\"`, anywhere in\n prose *before* the JSON will zero an otherwise perfect answer. Defensible given \"no prose,\n no code fences\", but it conflates ignoring the contract with writing a sentence.\n- **`validate()` checks shape, not truth.** It verifies the answer key is well-typed and\n that a strawman scraper does not solve the row — and generation already enforces the\n latter, so it cannot fire on a generated row. It is a regression guard on the generator,\n not evidence about the shipped rows. The reference-scraper check that *would* make it\n meaningful lives in `scripts/leak_scan.py`.\n- **The tiers do not order difficulty.** Hard rows always carry `variant_gallery`, and that\n pathology makes rows *easier* — the option-id attribute gives an explicit filter, so the\n strong arm scores 0.83 with it against 0.66 without. Read the aggregate, not the tiers.\n\n## How to read the score\n\nOne reward, `correct`, weight 1.0, binary. The aggregate is the solve rate. Weight-0\nmetrics carry the diagnostics: `field_misses` reports per-field hit/miss, which is what\nrevealed that an earlier build's entire difficulty lived in one field.\n\n### Baselines over the published 120 rows\n\n| baseline | solve rate | field accuracy |\n|---|---|---|\n| empty / unparsable / `{}` / all-null | **0.000** | 0.000 |\n| oracle per-field majority class | **0.000** | 0.13 |\n| **naive BeautifulSoup scraper** | **0.000** | **0.345** |\n| composite lazy scraper (three shortcuts) | 0.636 | — |\n| `deepseek-v4-flash` | 0.642 | — |\n| `openai/gpt-5-mini` | **0.767** | — |\n| honest semantic scraper | **1.000** | 1.000 |\n\nThe naive scraper is a deliberately hasty 30-line script. Per field it gets\n`title 58/120, price 22/120, currency 93/120, in_stock 64/120, sku 15/120,\nimage_urls 0/120, dispatch_date 38/120`.\n\n## Does it separate models?\n\nYes. 120 paired rows, same build, err 0.00 in both arms, sampling pinned:\n\n| | solve |\n|---|---|\n| `deepseek-v4-flash` | 0.642 |\n| `openai/gpt-5-mini` | **0.767** |\n\n20 strong-only, 5 weak-only discordant pairs, **exact McNemar p = 0.0041**. The arms are\nfrom different families, so this doubles as the generalisation check that reserved\npathology combinations cannot provide.\n\nWhere the difficulty sits:\n\n```\nfield failures flash gpt-5-mini\n image_urls 0.28 0.22\n title 0.15 0.03\n everything else <=0.02 <=0.01\n```\n\n## Running it\n\n```bash\nuv pip install -e envs/html_extraction\n```\n```bash\nuv run validate html-extraction --runtime.type subprocess -c 8\n```\n```bash\nset -a && . ./.env && set +a # the eval harness does NOT load .env itself\nuv run eval html-extraction --env.agent.harness.id null --env.agent.runtime.type subprocess \\\n -m -n 120 -r 1 -c 4 --sampling.temperature 0.0 --sampling.reasoning-effort medium\n```\n```bash\nuv run python scripts/compare_runs.py \nuv run python scripts/leak_scan.py --seeds 3\n```\n\n**Pin the sampling.** Reasoning effort is not fixed by default and the provider varies it\nbetween runs: one rollout in an unpinned run burned 131,076 reasoning tokens where the\npinned run maxes at ~3,000. Every comparison made before pinning had it as an uncontrolled\nvariable, and one of them produced a spurious null result.\n\n**Rate limits are per model.** `deepseek-v4-pro` returned 429 for every request while\n`deepseek-v4-flash` and `anthropic/claude-haiku-4.5` returned 200 from the same key in the\nsame minute. Errored rollouts do not bill, so a throttled run costs wall-clock, not money.\n\nKnobs: `--env.taskset.difficulty easy|medium|hard|all`, `--env.taskset.seed`,\n`--env.taskset.held_out true`, `--env.taskset.probe `,\n`--env.taskset.probe-n`.\n\n## Measuring difficulty instead of predicting it\n\n`--env.taskset.probe ` builds rows carrying one pathology **and nothing else**;\n`a+b` probes a combination; `none` is the clean floor. This exists because the project has\nrepeatedly been wrong about which content is hard — three pathologies added specifically to\nraise difficulty turned out to be the three weakest, and the change that actually worked\nwas a one-line edit nobody flagged.\n\n## The leak detector\n\n`scripts/leak_scan.py` is the tool that replaced hand-patching. For each answer field it\nenumerates every candidate the page offers, describes each with cheap features (document\nindex, DOM depth, child count, whether it carries a class or a style attribute, numeric\nrank, how often its id appears elsewhere), and asks whether any single feature picks the\nright candidate above chance **on seeds it was not fitted on**. It also runs an invariants\npass over properties true on nearly every row — **in both directions**, since \"almost\nalways false\" is as usable as \"almost always true\" — and the answerability check above.\n\nEvery rule is scored twice: over all rows, and over only the rows carrying the pathology\nthat makes the capability necessary. That distinction separates a leak from mere coverage,\nand it is the one this project kept getting wrong by hand.\n\n## Provenance\n\nEvery fixture is original synthetic HTML from a seeded template engine. No scraped pages,\nno copyrighted markup, no real brands. Reproducible from a seed and extensible without\ncollecting anything.\n\n## Known gaps\n\n- **A human has not read the rollouts.**\n- The residuals above, in particular the dispatch-date lookup table.\n- The tiers do not order difficulty.\n- Only two models, from two families, have been run against this.\n- `duplicate_gallery` reaches only 8/120 rows, squeezed out by its conflict with\n `variant_gallery`.\n","encoding":"utf-8","truncated":false,"total_bytes":14061},"status":null}