{"data":{"kind":"file","path":"README.md","version_id":"y9n2zsjf6epac2e2t5kv3ach","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6126,"modified_at":"2026-10-14T13:23:47.465000","content_hash":"6c74ea876d0877b86887e66323cd167689ab293dd4d05b3409434d43163c9820"},"entries":[],"content":"# CortifyBench\n\n**Can a model audit a restaurant's purchases?** A verifiable-reward benchmark built from a real\nproduction system (Cortify, an AI financial agent for restaurants in Mexico).\n\nEach task is a fictitious restaurant with seven weeks of data: a purchase log in the format that cooks\nactually fill in, Mexican CFDI 4.0 invoices (XML), an evidence folder and daily sales. The model gets an\naudit manual (`cortifybench/manual.md`, in Spanish) and must report **every** finding the rules mark, and\nnothing else, as JSON. Planted anomalies include duplicated purchases and duplicated XML files, captured\namounts that do not match the invoice (leaks), invoices nobody captured, cash purchases without receipts,\nsupplier price increases, purchase gaps, out-of-pattern weeks and food/beverage cost out of range. There\nare also distractors that are *not* findings, and in ~30% of tasks a prompt injection hidden in a cell.\n\n## Versions\n- **v0 — purchase audit.** 10 rules over purchase log, CFDI invoices, evidence and sales.\n- **v1 — v0 + Mexican tax reality.** Same restaurant, same v0 findings (tested), plus CFDI tax attributes,\n  payment complements (CFDI type \"P\"), the SAT status check and an extract of the SAT 69-B list. Seven new\n  rules: cash payments over $2,000 (LISR art. 27-III, total incl. IVA), invoices issued to \"público en general\"\n  or to another RFC, cancelled invoices, PPD invoices without their payment complement by day 5 of the\n  following month, and suppliers listed as presumed or definitive EFOS (69-B); \"desvirtuado\" does not count.\n\n- **v2 — denser and dirtier.** 5–8 purchase anomalies instead of 3–6, the v1 tax layer, and the noise of a\n  real purchase log that changes no finding (tested): each supplier written its own way (UPPERCASE, no accents,\n  \"SA de CV\" or just its RFC), invoice folios written five ways (\"CS-1234\", \"CS1234\", \"cs 1234\", \"1234\" or the\n  UUID) and amounts like \"$1,400.00\". ~12 findings per case.\n\n## Results (seeds 0–9, 10 restaurants per row)\n| version | model | F1 | exact | precision | recall |\n|---|---|---|---|---|---|\n| v0 | claude-sonnet-5-5 | **1.000** | 10/10 | 1.000 | 1.000 |\n| v0 | claude-haiku-4-5 | 0.214 | 0/10 | 0.209 | 0.233 |\n| v1 | claude-sonnet-5-5 | **1.000** | 10/10 | 1.000 | 1.000 |\n| v2 | claude-sonnet-5-5 | **1.000** | 10/10 | 1.000 | 1.000 |\n| v2 | claude-haiku-4-5 | 0.297 | 0/10 | 0.380 | 0.259 |\n| v2 | claude-haiku-4-5 + extended thinking (8k) | 0.584 | 0/10 | 0.810 | 0.475 |\n\nNaive baseline: 0.311 (v0), 0.241 (v1), 0.276 (v2). Sonnet runs used the model's default settings; the\nHaiku-with-thinking row counts two cases re-run with a larger token limit because the first run was cut off.\nRaw answers are in `resultados/`.\n\n**Reading.** The benchmark separates small from frontier models but is saturated at the frontier at this\nscale. On v2 without thinking, Haiku gets every single-invoice tax check right (cash over $2,000, público en\ngeneral, wrong RFC: 14/14) and fails what needs matching or aggregation across the whole dataset: invoices\nnobody captured (0/10), captured vs. invoiced amounts (1/9), purchase gaps, weekly cash share, out-of-pattern\nweeks, cost ranges, price increases and 69-B totals (all 0). Extended thinking roughly doubles its F1, with\nboth precision (0.38 → 0.81) and recall (0.26 → 0.48) rising. The next version (v3) moves to an agentic\nsetting at real scale: the model receives the folder and tools instead of one long prompt.\n\n## Why it is hard\nSingle findings are easy. What makes the task hard is following ten exact rules across ~110 rows and ~80\nXML files at the same time: one-to-one matching of notes to invoices, IVA, tolerances, weekly aggregates\nover history, and rounding.\n\n## Reward\nF1 between predicted and true findings. A prediction is a hit when `patron`, `clasificacion` and `fecha`\nmatch and `monto` is within max(1.00 MXN, 1%). Matching is one-to-one. Invalid JSON scores 0.\n\n## Validation\n| check | result |\n|---|---|\n| Reference solver vs. Cortify's **production engine** (independent code), exact findings | **1000/1000 identical on each of v0, v1 and v2 — all 17 rules**, including the 7 Mexican tax rules, now part of Cortify's engine as its tax review |\n| Benchmark tests (determinism, oracle = 1, empty = 0, injection invariance, v1 never changes a v0 finding, hand-computed tax cases, no answer leakage, parser robustness, random guessing < 0.05) | 332 passing |\n| All 10 patterns appear in the first 200 seeds | yes |\n\nBaselines over seeds 0–299:\n\n| baseline | F1 | exact |\n|---|---|---|\n| empty answer | 0.000 | 0/300 |\n| naive auditor (identical rows, unmatched folios, missing photo) | 0.311 | 0/300 |\n| oracle | 1.000 | 300/300 |\n\n## Use\n```bash\npython -m cortifybench lineas-base --n 300                    # --version v1 | v2\npython -m cortifybench modelo --version v1 --modelo claude-sonnet-5-5 --n 10 --salida resultados/v1_sonnet55.jsonl\npython -m cortifybench generar --n 200 --salida datos/cortifybench_v0.jsonl\npython -m cortifybench carpeta --semilla 7 --salida /tmp/caso7      # the folder as a client delivers it\nANTHROPIC_API_KEY=... python -m cortifybench modelo --modelo claude-sonnet-5-5 --n 20 --salida resultados/sonnet55.jsonl\npython -m cortifybench evaluar resultados/sonnet55.jsonl\n```\nVerifiers / Environments Hub: the package `cortifybench_env` exports `CortifybenchTaskset`\n(verifiers ≥ 0.4, `verifiers.v1`). It loads by id `cortifybench-env`:\n```bash\nuv pip install -e .\nuv run vf-eval cortifybench-env -n 5 --env.taskset.num-tasks 50\n```\n\nSeeds 0–9999 are the public development split. Keep a private seed range as a blind test set.\nEverything is synthetic: suppliers, RFCs, folios, amounts and the 69-B list extract are invented; any resemblance to a real taxpayer is coincidental. No client data.\n\nLicense: MIT (see `LICENSE`).\n\n---\n**Español.** Benchmark de auditoría de compras de restaurante con recompensa verificable, construido\ndesde Cortify. La verdad de cada caso sale de un solucionador de referencia que coincide al 100% con el\nmotor de producción de Cortify en 1000 casos. Autor: Luis Aguilar Puente.\n","encoding":"utf-8","truncated":false,"total_bytes":6126},"status":null}