{"data":{"kind":"file","path":"README.md","version_id":"eyelz56yps5bievvfq5lu566","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":24184,"modified_at":"2026-09-01T02:37:15.389000","content_hash":"5afacc1f8f439fca38ebad68929d9f5c4e3a1f08eba2bbf87b77a20c61aef542"},"entries":[],"content":"# event-recon\n\n**Environment ID:** `event-recon` · **Version:** 1.0.0 · **License:** MIT\n\n## Overview\n\nTwo production systems disagree about money. Find out why, and fix it.\n\nA creator-referral platform ingests webhook events from a partner brand. The platform's\ncommission ledger and the brand's own statement do not reconcile. The agent gets the\nplatform's database, the raw delivery stream, the brand's statement, and the integration\ncontract — and has to produce two things: a **findings report** naming each defect, the\npartners whose balance it moved, and an event that evidences it; and a **remediation\nscript** that, run against an unmodified copy of the database, lands the ledger exactly\non what the contract says is owed.\n\nThis is forensic accounting under an ordering-and-idempotency contract. It rewards\nreading a specification carefully, holding a multi-entity state model in working memory,\nand telling a real defect apart from a legitimate anomaly — and it punishes flagging\neverything, guessing, and brute force.\n\n**Tags:** `reconciliation` `forensic-accounting` `sql` `agentic` `tool-use`\n`data-integrity` `debugging` `verifiable-rewards` `procedural-generation` `long-horizon`\n\n### Why the ground truth holds up\n\n- **There is no answer file.** A task *is* a seed. The world, the injected defects, the\n  manifest and the correct ledger are all recomputed from that seed at scoring time.\n  Nothing is stored, so nothing can drift and nothing can leak.\n- **Two independent implementations.** The generator implements the accounting contract\n  forward; `core/reconciler.py` implements it backward, from `INTEGRATION.md` alone. They\n  share no accounting helper — the reconciler imports record types and the pinned zone\n  table, and derives every monetary quantity itself — and they must agree on every task or\n  the grader refuses to score it. The cross-check has already caught two real contract\n  defects that no test would have found.\n- **No LLM judge anywhere.** Every component of the reward is arithmetic over integers.\n\n## Datasets\n\n| Split | Tasks | Tiers | Availability |\n| --- | --- | --- | --- |\n| `public` | 120 | 30 each of T1–T4 | Published here and on the Hub |\n| `private` | 120 | 30 each of T1–T4 | Held back: unpublished seeds, plus three defect families absent from the public registry |\n\nEvery published task record carries a contamination canary\n(`event-recon-canary:dd06eda7-…`). It is deliberately absent from every agent-visible\nfile — a workspace containing it would teach the string during ordinary rollouts and\ndestroy its value as a tripwire. CI asserts both halves.\n\nTask worlds are generated, not authored. Per seed, the following are randomized: entity\nids, partner and program names, the partner brand, referral codes, dates, currency,\npayout timezones, program rates, qualification bars, accounting bases, order values, the\nenvelope-version cutover, five webhook payload field names, and two SQL column names.\nNothing transfers between tasks by memorization — SQL copied from another task's\ntranscript hits a missing-column error rather than a silently wrong answer.\n\n## Task\n\nThe agent's workspace holds exactly four files:\n\n| File | What it is |\n| --- | --- |\n| `db.sqlite` | The platform's system of record: `partners`, `programs`, `referral_codes`, `events_ingested`, `ledger_entries`, `payouts` |\n| `webhooks.jsonl` | Every envelope received, in delivery order, with receipt time, attempt number, backfill flag and signature |\n| `partner_statement.csv` | The brand's own summary — authoritative about volume, deliberately wrong about money |\n| `INTEGRATION.md` | The contract: 14 numbered accounting rules, 2 wire-typing rules, and the definition of a correct ledger |\n\nFive tools, and no shell: `sql_query` (read-only), `read_file`, `write_file`,\n`run_python` (stdlib only, no network, confined to the workspace), `submit_report`.\n\n### Defect families\n\nTwelve in the public registry, each with parameterized severity, split across where the\nfailure lives:\n\n*On the wire* — `duplicate_delivery`, `out_of_order_delivery`, `missing_field_coercion`,\n`schema_drift`, `currency_or_rounding_drift`, `partial_backfill`.\n*In reference data* — `identity_fragmentation`, `tier_misconfiguration`.\n*In the derivation* — `gross_net_divergence`, `timezone_boundary`,\n`retroactive_program_change`, `qualification_bar_mismatch`.\n\nThe platform's ledger builder carries seven latent bugs — dedup on `event_id` only,\ndelivery-order processing, dead-lettering orphaned effects, coercing an absent value to\nzero, taking a type violation literally, ignoring a currency mismatch, trusting its code\ncache. Each is **inert in a clean world** and only produces a wrong number once a wire or\nreference defect activates it. That is how production accounting systems actually fail.\n\n### Distractors are mandatory\n\nEvery task contains anomalies that look like defects and are not: genuine refunds\n(including instalments), delivery retries that repeat an `event_id`, complete backfill\nbatches, several genuinely distinct same-day orders from one buyer, a code that lawfully\nchanges program mid-window, orders that legitimately fall under a qualification bar, and\nauthorized manual adjustments that reconcile against no event at all. None appear in the\nmanifest, and a flag-everything policy loses precision faster than it gains recall.\n\n### Difficulty tiers\n\n| Tier | Shape |\n| --- | --- |\n| T1 | One defect, single-partner blast radius |\n| T2 | Two independent defects, distinct families |\n| T3 | Three to five interacting defects where remediation order matters |\n| T4 | T3 plus a family whose detection requires reasoning about idempotency, ordering or wire typing from the contract, plus the heaviest distractor density |\n\nSeven invariants are asserted before a generated task is allowed to exist. A task failing\nany of them is regenerated, not shipped. Seven is the count of the numbered ones, and\n`build_task` in fact rejects a plan on ten conditions — the three unlisted ones shape a tier\nrather than protect the reward (tier 1 must move exactly one partner, tier 2 and above must\ndraw distinct families, tier 4's defects must interact), and a task failing one of those is\nwell formed and merely belongs elsewhere:\n\n1. the clean world is genuinely clean — before injection the contract-faithful reading and\n   the platform's own ledger agree exactly, so the seven latent bugs really are inert;\n2. every injected defect moved at least one partner's balance;\n3. every defect leaves the *shipped* ledger visibly wrong for a partner it names, so\n   nothing in the answer key is unobservable from the workspace;\n4. every partner no defect touched already has a correct ledger, which is what makes the\n   non-destructiveness component well defined;\n5. the total blast radius stays at or under 1.0 `(family, partner)` pairs per partner\n   (`MAX_PAIRS_PER_PARTNER` in `core/generator.py`) — a task where the defects between\n   them move most of the roster is a systemic outage, and it is also the shape that makes\n   brute-force enumeration profitable;\n6. **no task is solvable by copying the counterparty's numbers.** The brand's statement\n   normally disagrees with what is owed, but \"normally\" is not good enough: one private\n   task reached 0.30 ledger correctness by pasting their totals in before this check\n   existed;\n7. **most of what a task breaks needs the contract to repair.** Some defects are fixable\n   as bookkeeping — drop the rows keyed to a superseded event, drop an amendment that\n   carries no gross field, re-key an absorbed partner, re-bucket a period — and no rate,\n   basis or rounding mode is involved in any of them. At most half of a task's broken\n   partners may be repairable that way, which bounds what a policy that never opens\n   `INTEGRATION.md` can score on the ledger component.\n\n   The bound is on the *task*, not on the family, because repairability is a property of\n   the defect and not of its name: one family is bookkeeping in one of its two modes and\n   accounting in the other. An earlier version of this check tested family names against\n   a fixed list and read as a stronger guarantee than it was — a task mixing a\n   bookkeeping defect with an accounting one that happened to break no partner slipped\n   through and was finishable for free.\n\n## Quickstart\n\n```bash\nuv pip install -e .\n\n# The whole gate suite, no model and no network required.\npython -m event_recon.adapters.cli gates\n\n# Inspect one task's workspace, exactly as an agent would see it.\npython -m event_recon.adapters.cli materialize --seed 4100000 --tier 3 --out /tmp/ws\npython -m event_recon.adapters.cli solve --seed 4100000 --tier 3   # gold trajectory\n\n# Evaluate a model. The tool-only harness is the package default. This is the one\n# step that needs a provider key: copy `.env.example` to `.env`, fill it in, and\n# export it -- nothing here loads a `.env` on your behalf.\n#   set -a; source .env; set +a\nuv run eval event-recon -n 10 -r 3 -m <model-id>\n\n# A held-back split, if you have one: its module supplies both the extra defect\n# families and its own seeds, so nothing about it needs to exist in this package.\nuv run eval event-recon --env.taskset.family-module <module> -n 10 -m <model-id>\n\n# The oracle gate as a framework operation, no model needed. The runtime flag is\n# required: verifiers defaults `validate` to the Prime runtime, which wants an API key,\n# and nothing in this environment needs a container.\nuv run validate event-recon --only-gold --runtime.type subprocess\n```\n\nOnly the three `uv run` lines need `verifiers` installed. Everything else — generation,\ngrading, the oracle, the probes, the whole gate suite — runs on a stock interpreter with\nno third-party package at all, which CI asserts on every push.\n\nThe package declares one further runtime pin, `tzdata`, and it is not a convenience: rule\nR9 buckets every payout period through a zone lookup, so the IANA database is an input to\nthe graded answer and is pinned rather than inherited from the host. It is a soft import —\nwithout it `core/zones.py` falls back to the host's database and `zones.source()` reports\nthat it did, which is why the core still runs with nothing installed.\n\n## Taskset Config\n\nUnder `--env.taskset.*`:\n\n| Key | Default | Meaning |\n| --- | --- | --- |\n| `tier` | `None` | Restrict the split to one difficulty tier |\n| `num_tasks` | `None` | Take the first N tasks after the tier filter |\n| `family_module` | `None` | Import path of a held-back split: supplies its extra defect families *and* its seeds |\n| `task.script_timeout_seconds` | `90.0` | Wall-clock limit on the graded remediation script |\n| `task.max_turns` | `80` | Hard turn cap for a rollout that never submits |\n| `task.tools.sql_row_limit` | `200` | Rows one `sql_query` may return |\n| `task.tools.sql_response_bytes` | `1000000` | Bytes one `sql_query` response may carry |\n| `task.tools.sql_timeout_seconds` | `10.0` | Wall-clock limit on one `sql_query` |\n| `task.tools.read_bytes_limit` | `120000` | Bytes one `read_file` may return |\n| `task.tools.python_timeout_seconds` | `30.0` | Limit on an exploratory `run_python` |\n\n`sql_response_bytes` is a byte bound because the row bound is not one: SQLite builds a cell\nof any width on request, so 200 rows — inside the row cap, nothing reported as truncated —\nwere measured at 152.6 MiB. The default is 8.9× the largest response any legitimate query\nproduced across all 30 tier-4 tasks. A truncated response says so, in `truncated` and\n`truncated_by`. The same defaults and the same truncation rule back both harness surfaces;\nthey live in `event_recon/toolsets/limits.py`, which imports no harness package so the\nOpenEnv adapter can share them.\n\n```bash\nuv run eval event-recon --env.taskset.tier 4 -n 30 -r 5\n```\n\n## Harness Config\n\nThe package exports `EventReconHarness`, a tool-only MCP loop, and it is the taskset's\ndefault. This is deliberate: verifiers' built-in fallback is `bash`, which on a task\ndeclaring `NEEDS_CONTAINER = False` hands the rollout a shell on the **host** — outside\nthe audited sandbox, outside the workspace, with this package importable. The five\nworkspace tools are the entire intended contract.\n\nAny MCP-capable harness works if you want one: `--env.agent.harness.id <name>`. Nothing\nin the environment needs a container, a GPU, or the network; the whole gate suite runs on\na laptop.\n\nResource limits declared per rollout: the graded remediation script gets 90 s wall clock,\n60 s CPU, a bounded `RLIMIT_FSIZE`, `RLIMIT_CORE = 0`, and a scrubbed environment with\n`sys.path` stripped to interpreter-owned stdlib entries. It cannot reach the network,\nspawn a process, read outside its workspace, or import this package.\n\n\"Cannot spawn a process\" is enforced by the audit hook alone, and that is a correction:\nthis paragraph used to also claim `RLIMIT_NPROC = 0`. That limit was removed after\nmeasurement showed it denied nothing the hook did not already deny — `os.fork`,\n`subprocess`, `multiprocessing`, `_posixsubprocess` and `os.posix_spawn` all return the\nhook's own refusal with it and without it — while making a score depend on the host, since\nLinux counts threads against `NPROC` and macOS counts only processes. A threaded script\npassed on one and failed on the other.\n\n## Metrics\n\nReward is `0.50·ledger + 0.30·identification + 0.10·evidence + 0.10·preservation`,\nmultiplied by a validity gate that is zero unless a report was submitted, at least one\nfinding was made, and the remediation script ran cleanly inside the sandbox.\n\n| # | Component | Weight | Definition | What stops it being gamed |\n| --- | --- | --- | --- | --- |\n| 1 | `ledger_correctness` | 0.50 | Fraction of the partners whose ledger was actually broken that now match the contract's balance projection exactly, at zero tolerance | Recomputed from raw events by an independent reconciler, never diffed against a stored file. Collapses to 0 if any already-correct partner regresses. Copying the partner statement scores 0, because the statement is computed under a documented, different policy |\n| 2 | `defect_identification` | 0.30 | Fβ (β = 0.5) over `(family, partner_id)` pairs against the injection manifest | Entity ids are required, so reciting the published family list scores nothing. Distractors punish over-flagging |\n| 3 | `evidence_grounding` | 0.10 | Fraction of *claimed* `(family, partner)` pairs whose cited `event_id` genuinely evidences that defect | Denominated over every claim, not just correct ones, so claiming widely and citing loosely costs twice. Keyed by distinct pair, so repeating a grounded claim is inert — measured at 0.10 → 0.96 from repetition alone before that was closed. The manifest's evidence is narrowed at generation time to events on orders the defect actually moved |\n| 4 | `non_destructiveness` | 0.10 | Whether partners that needed no repair came out value-identical, and no `payouts` row changed | **Scaled by** ledger correctness: \"I broke nothing\" is a property of every empty submission. Scaled rather than gated, because a step function at the first repaired partner made the first repair worth ~5× the second |\n\n**One deliberate divergence from the specification this was built to.** Component 2 uses\na precision-weighted Fβ with β = 0.5, not the symmetric F1. The contract has to publish\nthe list of recognized families — an agent cannot name what it found otherwise — so\nrecall is purchasable by brute force: cross every family with every partner and recall is\n1 by construction. Measured under F1 that policy peaked at **0.060**, above the 0.05\nadversarial ceiling, and no reachable roster size fixes it because F1 rises with recall\nfaster than a growing claim space drops precision. Under β = 0.5 its worst case over all\n120 published tasks is **0.0389**, mean 0.0185 — measured against the *strengthened* fixture that cites every event on the wire, not the token five it cited before.\nPrecision-weighting is also the operationally correct incentive here: naming a partner\nyou cannot evidence sends money or an investigation to the wrong person. Plain F1,\nprecision and recall are all logged as zero-weight metrics, so the symmetric number is\nrecoverable from any trajectory file.\n\nZero-weight metrics logged per episode: turn count, per-tool call counts, input and\noutput tokens, wall time, SQL query count, remediation script length in bytes and lines,\nsandbox violations, partners total / broken / repaired / regressed, precision, recall,\nplain F1, and a `failure_code` index into a ten-label failure taxonomy.\n\n### Gate results\n\nMeasured over the full 120-task public split, no sampling:\n\n| Gate | Result |\n| --- | --- |\n| Oracle | 1.0000 on every task; 0 failures over 200 re-runs |\n| Null policy | 0.0000 mean and max |\n| **Strongest workspace-only policy** | mean **0.3421**, max 0.5500, **0 of 120 solved**, ledger component 0.0640 |\n| Single-axis probes (`CHEAT_PROBES`) | all ≤ 0.05; worst `flag_everything` at 0.0389 |\n| Concentrated report sweep | `flag_one_family` at 0.2293 max, gated at its own `SWEEP_CEILING` (0.32, derived over both splits) rather than at `CHEAT_CEILING` — the two assert different things |\n| Generator ↔ reconciler cross-check | 120/120 tasks, 3055 partner projections |\n| Workspace determinism | 120/120 byte-identical |\n| Same-trajectory re-scoring | one distinct reward value over 100 re-scorings |\n| Ground-truth leakage | 0/120 workspaces, by string scan, column scan *and* derived-feature scan |\n| Canary | on 120/120 records, in 0/120 workspaces |\n\n**Read the third row before the fourth.** The 0.05 ceiling bounds the probes that each\nattack one component in isolation; it is *not* a bound on adversarial reward, and treating\nit as one would be the most misleading number in this README. The concentrated report sweep\nin the row below it is gated at its own, higher ceiling for exactly that reason -- the two\nceilings assert different things, and only the composite bounds adversarial reward. `event_recon/core/attacks.py`\nships the strongest policy we could build that never reads `INTEGRATION.md` and computes no\ncommission: it recovers defects by syntactic pattern-matching, and it repairs whatever can be\nrepaired by deleting a row, re-keying a partner or re-bucketing a period. It earns most of\nthe identification and evidence weight, a small fraction of the ledger, and it never solves a\ntask. That gap is the environment's actual claim: the report is worth 0.40 and is partly\nheuristic; the repair is worth 0.50 and is not.\n\nThat number is a **floor on the attack, not a ceiling**, and this README said so for eight\nversions while understating how far the floor was from the ceiling. The attack repaired two\nfamilies. Three more were repairable by the same kind of statement and nobody had written\nthem: delete the rows of an amendment that carries no gross field, delete the rows of an\norder billed in a currency the program does not pay in, and recompute every row's payout\nperiod from its program's own time zone. Roughly forty lines, no rate, no basis, no rounding\nmode, no contract read.\n\nAdded, the same policy went from 0.0530 of the ledger component to **0.2065**, and from\nsolving nothing to **solving 12 of the 120 tasks outright** — every one of them a tier-1\nworld whose only defect was a missing field, on the tier this document had described as the\nsafest. Both composite bounds were breached. The environment's central claim, that the\nrepair half is not heuristically reachable, was false for as long as the attack was the one\nthat had been written rather than the one that could be.\n\nThe generator was rebuilt in response rather than the number restated: invariant 7 now\nbounds the share of a task's broken partners that need no contract, so the bound holds\nagainst the stronger attack instead of against the weaker one. The three repairs ship, in\n`event_recon/core/attacks.py`, because an adversarial ceiling nobody can reproduce is\nmarketing. The next person to widen this attack should expect to find something too, and\n`tests/test_gates.py` fails the moment they do without telling `core/defects.py`.\n\nThe private split passes the same eight gate families, run through the shipped adapter with\n`--env.taskset.family-module`: oracle 1.0000, null 0.0000, worst single-axis probe 0.0367,\nworkspace-only policy max 0.6813 with a 0.0839 ledger component, **0 of 120 solved**.\n\n**Two of those bounds were re-derived to get there, and that is worth stating plainly.**\nWhen the private seeds were rotated — they had been an arithmetic progression, which is a\nseparate story — every private world changed, and two ceilings that had been fitted to\nmeasurements taken on the *public* split at generator 1.9.0 turned out not to hold on the\nnew draw. The concentrated sweep reached 0.2897 against a 0.25 ceiling, on 3 of 240 tasks\nacross both splits; and the tier-3 mean of the composite's ledger component reached 0.1542\nagainst a 0.15 line that had been introduced for the split-wide mean and then reused per\ntier without anyone deriving it for that scope.\n\nBoth were re-fitted over all 240 tasks, unsampled, and both now name the sample they came\nfrom in their own docstrings: `SWEEP_CEILING` is 0.32 and the per-tier check has its own\nconstant, `COMPOSITE_MAX_TIER_LEDGER`, at 0.17. The honest reading is that a number fitted\nto one sample met a different one, not that the environment got worse — the private split's\nsweep *mean* is lower than the public split's (0.1023 against 0.1162), the two distributions\nagree where they are dense, and the difference is a right tail three tasks long. Nothing is\nsolved on either split, and the split-wide ledger means — 0.0640 public and 0.0839 private —\nare both well inside the 0.15 line that was never in question.\n\nWhat has not changed is which numbers are load-bearing. `SWEEP_CEILING`'s own docstring\ncalls it a drift detector rather than a security bound, and the thing that actually bounds\nadversarial reward is the composite, which is gated on a per-task maximum that no\nre-derivation touched: no workspace-only policy may finish a task, on either split.\n\nOne earlier figure on this page was not stale but wrong. The private ledger component was\npublished as 0.0430, which was a mean pooled across *both* composite probes by a gate runner\nthat has since been deleted — and `report_only_policy` ships a no-op script, so it scores\nzero and halved the number. The comparable figure is 0.0839.\n\n`reports/` has the procedure behind each number and an honest account of what is *not*\nmeasured: `datasheet.md`, `difficulty_calibration.md`, `determinism.md`, `api-notes.md`.\n\n**Not measured: frontier-model pass@1.** The sweep harness is written\n(`scripts/calibrate.py`) and the difficulty band is a design target rather than a result\nuntil it runs at scale. `reports/difficulty_calibration.md` says so in its first paragraph.\n\nWhat *has* run is its smoke line — one model, one task per tier, one rollout — and it is\nworth the sentence because it is the first time this environment has been driven by a live\nmodel end to end. Three of four rollouts played to a submitted report and the fourth was\naborted by a provider rate limit; across the three, identification scored 0.714, 0.361 and\n0.000 while **ledger correctness scored 0.000 on all three**. Four rollouts prove nothing\nabout difficulty. They do show the shape this README argues for: the report half is partly\nreachable and the repair half was not reached at all.\n\nGetting there took four fixes to the harness, and none of them was cosmetic. Three were\nfound by reading it — a client flag that does not exist, a trace parser reading the wrong\nlevel of the schema so every reward came back 0.0, and a run-directory collision that\nmislabelled tiers. The fourth was found by running it: `eval` takes its runtime as\n`--env.agent.runtime.type`, not the `--runtime.type` that `validate` takes, so every rollout\ntried to provision a billed cloud sandbox, failed with `Payment required`, made zero model\ncalls, and reported pass@1 = 0.0 on every tier. An unrun script is not evidence of\nanything.\n","encoding":"utf-8","truncated":false,"total_bytes":24184},"status":null}