{"data":{"kind":"file","path":"README.md","version_id":"ss2nqmrxx2tnq6xflrprqemr","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":8001,"modified_at":"2026-08-01T18:24:16.818000","content_hash":"ca7c00aeb9f0d55c721f28a99c57150b2b543b8a51042eb3ee2bd0576a5591b5"},"entries":[],"content":"# dirty-integration\n\nAn RL environment for [verifiers](https://github.com/PrimeIntellect-ai/verifiers).\nThe agent has to collect every record from a paginated third-party API. The API\nmisbehaves in the four ways real ones do, and none of them announce themselves.\n\nSource: [github.com/buildok/dirty-integration](https://github.com/buildok/dirty-integration)\n\n## Why this one\n\nExisting agent benchmarks for business workflows — [AutomationBench](https://app.primeintellect.ai/dashboard/environments/zapier/automationbench)\namong them — measure whether an agent drives a process correctly across tools\nthat work. Their traps are data traps: a stale record, an irrelevant row.\n\nThis one measures the other axis. The process is trivial; the tools lie.\n\nAnyone who has shipped an integration knows the difference. The happy path is\nan afternoon. The fortnight goes on the vendor whose cursor occasionally\nrepeats, whose `total` was never accurate, and whose 429 arrives without a\n`Retry-After` header.\n\n## What the API does\n\n| Fault | What the agent sees |\n|---|---|\n| **Cursor repeat** | `next_cursor` sometimes points back at the page just served. Following it blindly loops and fills the result set with duplicates. |\n| **Inflated total** | `total` is larger than the real record count. An agent that pages until `len(collected) == total` never terminates. |\n| **Type drift** | `amount` arrives as `int` on some pages and `str` on others, for the same schema. |\n| **Timeout** | The request dies with no response — and still spends quota, as real gateways do. |\n| **Request quota** | `429` with no `Retry-After`, and **it never resets**. Retries are paid for out of the same pocket as progress. |\n\nThe quota is a multiple of the pages actually needed (`2.0×` on `realistic`,\n`2.2×` on `hostile`), not a fixed number. A hard constant would silently become\nan impossible task the moment someone doubled `num_records`.\n\nThree profiles: `clean`, `realistic`, `hostile`. `clean` is a working API and\nexists as a baseline — an agent that cannot score 1.0 on `clean` has a problem\nunrelated to fault handling.\n\n## Scoring\n\nThree separate rewards, because \"did it work\" and \"did it behave\" are different\nquestions and averaging them hides the answer.\n\n| Reward | Weight | |\n|---|---|---|\n| `completeness` | 0.6 | Jaccard of submitted ids against truth, minus a duplicate penalty. Recall alone would reward submitting every page seen, duplicates included — exactly what the cursor fault provokes. |\n| `type_handling` | 0.2 | Fraction of submitted rows whose `amount` is an `int` and matches the true value. `bool` does not count, despite being an `int` subclass in Python. |\n| `budget_discipline` | 0.2 | `1 / (1 + times_429)`. Not binary: one 429 while probing for an unpublished quota is reasonable, ten means the agent kept calling after being told to stop. |\n\n## Calibration\n\nThree reference solvers ship with the tests. They are not clever — they are the\nobvious strategy, the impatient one, and the careful one — and they exist so the\ndifficulty curve is a measurement rather than a hope.\n\nMean weighted reward, 60 seeds per cell:\n\n| Profile | naive | stubborn | careful |\n|---|---|---|---|\n| `clean` | 1.00 | 1.00 | 1.00 |\n| `realistic` | 0.56 | 1.00 | 1.00 |\n| `hostile` | 0.35 | 0.91 | 1.00 |\n\n- **naive** — follow `next_cursor`, keep everything, stop at the first failure.\n- **stubborn** — deduplicate and retry every failure immediately, forever.\n- **careful** — deduplicate, detect a repeated cursor, retry, coerce types,\n  and stay inside the quota.\n\nWhat the table establishes: the task is solvable (careful reaches 1.0\neverywhere), `clean` is winnable by the obvious approach, the faults punish\nnaivety, and `budget_discipline` is load-bearing rather than decorative — the\nstubborn solver collects nearly all the data and still loses 0.09 to it.\n\nIf a change ever pushes the careful solver below 1.0, the environment became\nunfair rather than harder. That is what `tests/test_solvable.py` guards.\n\n### Measured against real models\n\nMean weighted reward, 5 seeds per cell, `max_turns=30`, run through Prime\nInference on 2026-08-01. Total spend for the table: **$1.38**.\n\n| Model | `clean` | `realistic` | `hostile` |\n|---|---|---|---|\n| `google/gemini-2.5-flash-lite` | 0.00 | — | — |\n| `deepseek/deepseek-v3.2` | 1.00 | 1.00 | 0.76 |\n| `anthropic/claude-sonnet-5` | 1.00 | 1.00 | 0.81 |\n\nThree things this says, and one of them is unflattering:\n\n**`clean` and `realistic` do not discriminate between competent models.** Both\nscore a flat 1.00. If you are comparing models, run `hostile`; the other two are\nuseful as a floor, not as a benchmark.\n\n**`gemini-2.5-flash-lite` scored zero by never calling a tool at all.** It\nanswered in prose. That is a real result rather than a harness bug — the tool\ndefinitions reach the model, as the other rows show — but it means the\nenvironment cannot rank models below a certain tool-use competence. It only\ntells you they are below it.\n\n**On `hostile`, the two competent models differ almost entirely in discipline.**\nCompleteness and type handling are identical (0.80 each); the whole gap is\n`budget_discipline`, 0.59 against 0.83. The stronger model did not collect more\ndata — it wasted fewer requests getting it. That split is the reason the reward\nis three numbers rather than one.\n\nBoth models failed the same 1 seed in 5, hitting `max_turns` rather than\nanswering wrongly. The careful reference solver clears every one of those seeds\nin 11–16 requests of its 22-request quota, so the seed is not unwinnable — the\nmodels looped. Raising `max_turns` above 30 would likely raise both scores;\nthe figures above are for 30.\n\n## Determinism\n\nEvery fault is a pure function of `(seed, request_index)` — hashed, not drawn\nfrom an RNG. A shared RNG would couple the faults together, so one extra retry\nwould shift every later fault in the episode and the same seed would stop\nreplaying the same run.\n\nA reward that moves between runs is not a reward. It is noise, and an agent\ntrained against noise learns nothing.\n\n## Usage\n\n```bash\nuv pip install -e \".[dev]\"\npython -m pytest tests -q          # 43 tests, no model required\n```\n\n```python\nimport verifiers as vf\n\nenv = vf.load_environment(\n    \"dirty-integration\",\n    profile=\"realistic\",   # clean | realistic | hostile\n    num_tasks=12,          # one seed per task\n    num_records=40,\n    page_size=4,           # defaults give 10 pages, so faults fire reliably\n    max_turns=30,\n)\n```\n\n`num_records=40, page_size=4` is not arbitrary. An earlier default of 20 records\nover 5-record pages meant a 4-request episode, and a third of `hostile` seeds\nthen produced no timeout at all — the profile was hostile in name only.\nProbabilistic faults need enough draws to be reliable.\n\n## Where it breaks\n\nStated up front, because an environment whose limits are undocumented is one\nwhose scores cannot be interpreted.\n\n- **On `hostile`, the careful and stubborn request distributions overlap.** A\n  careful agent unlucky with timeouts spends nearly as much quota as a stubborn\n  one that got lucky. This is not tuned away: removing the overlap would mean\n  setting a quota that punishes honest retrying.\n- **The API is in-memory, not over a socket.** Timeouts are raised, not waited\n  out. Nothing here measures whether backoff is well-tuned in wall-clock terms —\n  only whether the agent stops.\n- **`budget_discipline` cannot distinguish a deliberate probe from carelessness.**\n  One 429 scores 0.5 either way.\n- **Type drift covers one field.** Real drift arrives in nested structures too,\n  which this does not model.\n- **No auth expiry, no 500s, no malformed JSON.** Those failures are loud: the\n  agent notices immediately and the episode ends. The four modelled here are\n  quiet — each one lets the agent believe it succeeded, which is the point.\n\n## Licence\n\nMIT. Built with AI assistance; the design decisions, the failure modes chosen,\nand the review are mine.\n","encoding":"utf-8","truncated":false,"total_bytes":8001},"status":null}