{"data":{"kind":"file","path":"README.md","version_id":"zdjxl6l1nkroehkl7cyottk7","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6311,"modified_at":"2026-09-19T23:12:11.338000","content_hash":"5cb86a9d45a958c14b1f5bd7511ac0bce76cb93e7d9dea5ee787f30842bc34f4"},"entries":[],"content":"# aml-alert-triage\n\nFirst-line anti-money-laundering alert triage. The model opens a transaction-monitoring\nalert the way an analyst in a bank's financial crime unit does — the rule that fired, a\nsubject profile, a transaction window — and must decide whether to **escalate** it to\ninvestigation or **dismiss** it as a false positive, naming the laundering typology when\nit escalates.\n\n### Overview\n\n- **Environment ID**: `aml-alert-triage`\n- **Type**: single-turn\n- **Tags**: finance, compliance, aml, financial-crime, classification, reasoning\n- **Data**: fully synthetic, procedurally generated from a seed\n\n> **No real data.** Every alert is generated procedurally. No real customer, account, or\n> transaction data is used, derived from, or reproduced, and no institution's detection\n> logic is described. Jurisdiction names in the high-risk set are deliberately fictional\n> so the task tests reasoning rather than recall of real country lists.\n\n### Why this task is hard\n\nThe interesting difficulty in AML triage is not spotting laundering. It is that **most\nalerts a monitoring system produces are false positives** — legitimate activity that\nhappens to trip a rule — and the analyst's job is to tell them apart under an asymmetric\ncost function.\n\nSo every benign case in this environment is adversarial by construction. Each one fires a\ngenuine monitoring rule for a legitimate reason:\n\n| Benign case | Fires | Why it is not laundering |\n|---|---|---|\n| Property sale settling into a purchase | same-day pass-through | Both counterparties are regulated: conveyancing escrow in, estate agent client funds out |\n| Restaurant / salon daily takings | repeated cash deposits | Irregular amounts, evening timing, consistent with declared turnover |\n| Salary swept to own savings | same-day pass-through, high outflow ratio | Named employer in, own-name account out, repeats monthly on the same date |\n| Insurance settlement | large round credit | Settlements are round by construction; named insurer, consistent follow-on spend |\n| Estate distribution | credit inconsistent with income | Distributions are expected to dwarf earnings; counterparty is a registered estate |\n\nA model that has learned \"large + round + fast = suspicious\" scores at the degenerate\nbaseline. Separating these requires reading the counterparty, the timing, and the\nsubject's declared profile together.\n\n### Typologies\n\n`structuring` · `layering` · `rapid_movement` · `round_amount` · `sanctions_nexus` · `none`\n\nThe monitoring threshold throughout is SEK 150,000, standing in for a large-transaction\nreporting trigger; the structuring generator places deposits just beneath it.\n\n### Scoring\n\n`triage_reward` (weight 1.0) is weighted by how the error types actually differ in cost:\n\n| Ground truth | Model | Score | Rationale |\n|---|---|---|---|\n| benign | dismiss | **1.0** | correct |\n| benign | escalate | **0.3** | false positive — costs analyst time, tolerable |\n| genuine | escalate, right typology | **1.0** | correct |\n| genuine | escalate, wrong typology | **0.6** | the escalation is the decision that matters |\n| genuine | dismiss | **0.0** | missed escalation — a regulatory failure |\n| — | unparseable | **0.0** | |\n\n`format_reward` (weight 0.1) checks the response is parseable.\n\nReported as metrics (weight 0.0): `false_negative`, `false_positive`, `escalation_rate`,\n`typology_accuracy`. `escalation_rate` exists specifically to expose the\nescalate-everything policy, which asymmetric scoring would otherwise partially reward.\n`typology_accuracy` counts `none` on a benign alert as correct, so it is an accuracy over\nthe whole population rather than a figure capped at the genuine-alert rate.\n\n### Reference policies\n\nSynthetic policies at `benign_rate=0.5`, n=600 — the floor any real result must clear:\n\n| Policy | triage_reward |\n|---|---|\n| perfect oracle | 1.000 |\n| right call, wrong typology | 0.813 |\n| **always dismiss** | **0.532** |\n| **always escalate** | **0.482** |\n| unparseable output | 0.000 |\n\nBoth degenerate policies sit at ~0.5, so anything below ~0.55 carries no triage signal.\n\n### Measured baseline\n\n`qwen2.5:7b-instruct`, n=50, `benign_rate=0.5`, temperature default:\n\n| Metric | Value |\n|---|---|\n| triage_reward | **0.796** |\n| typology_accuracy | 0.600 |\n| format_reward | 0.880 |\n| escalation_rate | 0.680 |\n| false_negative | **0.020** — missed 1 of 23 genuine alerts |\n| false_positive | **0.240** — over-escalated 12 of 27 benign alerts (44%) |\n\nThe error profile is the interesting part, and it is the profile a cautious junior analyst\nproduces: near-perfect recall on genuine alerts, bought with a 44% false-positive rate on\nthe adversarial benign cases. The benign alerts it escalated were concentrated in exactly\nthe two designed to be hardest — `funds_pass_through_same_day` (property sale, salary\nsweep) and `credit_inconsistent_with_income_profile` (estate distribution).\n\nTypology accuracy on genuine alerts breaks down sharply:\n\n| Typology | Accuracy |\n|---|---|\n| structuring | 1.00 |\n| layering | 1.00 |\n| rapid_movement | 1.00 |\n| round_amount | 1.00 |\n| **sanctions_nexus** | **0.00** |\n\nFour of five typologies are near-solved by a 7B model; `sanctions_nexus` is not recognised\nat all. That gap is a direct consequence of the fictional jurisdiction names — the model\ncannot pattern-match a country it has memorised as high-risk, and must instead reason from\nthe enhanced-due-diligence framing given in the alert. It does not.\n\n### Arguments\n\n| Arg | Default | Meaning |\n|---|---|---|\n| `num_alerts` | `200` | training split size |\n| `eval_alerts` | `60` | eval split size |\n| `benign_rate` | `0.5` | fraction of alerts that are false positives |\n| `seed` | `0` | generator seed; eval split uses `seed + 10000` |\n\n`benign_rate` defaults to 0.5 so a dismiss-everything policy cannot outscore genuine\ntriage. **Real first-line alert populations run above 0.9** — set `benign_rate=0.95` to\nmeasure behaviour under a realistic base rate, where the degenerate baseline rises and\nfalse-negative cost dominates.\n\n### Quickstart\n\n```bash\nprime env install fredrik/aml-alert-triage\nuv run vf-eval aml-alert-triage -n 20 -r 3\n```\n\nRealistic base rate:\n\n```bash\nuv run vf-eval aml-alert-triage -n 40 -r 3 -a '{\"benign_rate\": 0.95}'\n```\n\n### License\n\nMIT\n","encoding":"utf-8","truncated":false,"total_bytes":6311},"status":null}