{"data":{"kind":"file","path":"README.md","version_id":"v5qqdhp28lry5bd9r6rljbd8","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2248,"modified_at":"2026-03-23T14:55:54.865000","content_hash":"e96c05e443666194d8de3bcd147eb1585fe5665f51cc651d5eb3ae59e3a779ee"},"entries":[],"content":"# email-to-cc-bcc\n\n### Overview\n\n- **Environment ID**: `email-to-cc-bcc`\n- **Short description**: Given a multi-turn email thread with a roster of people, roles, and privacy constraints, assign recipients to To, CC, and BCC fields for each email.\n- **Tags**: structured-output, rules, email, train, eval, reasoning\n\n### Datasets\n\n- **Primary dataset(s)**: `gustofied/email-to-cc-bcc` (Hugging Face)\n- **Split sizes**: ~5000 rows (train), 91 rows (test)\n\n### Task\n\n- **Type**: multi-turn (1–3 turns, configurable)\n- **Output format**: JSON with exactly `to`, `cc`, `bcc` keys, each an array of email addresses\n- **Rubric overview**: Jaccard overlap per field per turn, averaged. Weighted sum with format scaffolding.\n\n### Quickstart\n\n```bash\nprime eval run email-to-cc-bcc -m gpt-4.1-mini -n 20 -r 3\n```\n\nSingle-turn on test set:\n```bash\nprime eval run email-to-cc-bcc -m <model> -n 50 -r 3 -a '{\"max_turns\": 1, \"dataset_split\": \"test\"}'\n```\n\n### Environment Arguments\n\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `dataset_name` | str | `gustofied/email-to-cc-bcc` | HuggingFace dataset path |\n| `dataset_split` | str | `train` | Dataset split (`train` or `test`) |\n| `max_turns` | int | `3` | Number of turns per conversation (1, 2, or 3) |\n\n### Metrics\n\n| Metric | Weight | Meaning |\n| ------ | ------ | ------- |\n| `reward` | - | Weighted average of all metrics below |\n| `to_correct` | 0.40 | Jaccard overlap for To field, averaged across turns |\n| `cc_correct` | 0.40 | Jaccard overlap for CC field, averaged across turns |\n| `bcc_correct` | 0.10 | Jaccard overlap for BCC field, averaged across turns |\n| `format_correct` | 0.05 | Binary: valid JSON with exactly {to, cc, bcc} keys and list values |\n| `email_format` | 0.05 | Fractional: proportion of recipients that are email addresses |\n\n### Notes\n\n- BCC ground truth is empty on ~60% of turns. Empty/empty scores 0.2 (small credit), non-empty when GT is empty scores 0.0 (penalizes hallucination).\n- `format_correct` and `email_format` are scaffolding rewards. They help small models learn output format before recipient placement.\n- The `test` split (91 rows) was generated independently from the training data using the same pipeline and validator.\n","encoding":"utf-8","truncated":false,"total_bytes":2248},"status":null}