{"data":{"kind":"file","path":"README.md","version_id":"yls1qdr6g7fpzekxa984zzbl","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":9782,"modified_at":"2026-05-19T15:25:03.389000","content_hash":"8fc0f87c2b4227b93cfce21aade76da7b4a5fe1e4c9580956d059a949ca700b1"},"entries":[],"content":"# legal-cls\n\nClassify whether a user in a conversation is seeking legal guidance (`YES`/`NO`), and if so assign one legal topic from the taxonomy. Single-turn environment built on [Prime Verifiers](https://github.com/PrimeIntellect-ai/prime-verifiers).\n\n## Topic taxonomy (v2)\n\nFourteen labels, kept compact enough for training but more separable than the legacy set:\n\n| Label | Short description |\n| ----- | ----------------- |\n| `FAMILY` | Marriage, divorce, custody, support, adoption, guardianship, domestic violence. |\n| `HOUSING` | Rent, eviction, deposits, mortgages, foreclosure, habitability, neighbors, housing subsidies. |\n| `WORK` | Employment contracts, wages, dismissal, discrimination at work, leave, workplace safety. |\n| `PUBLIC_BENEFITS` | Unemployment, disability, pensions, welfare, eligibility, appeals on benefits. |\n| `CRIMINAL_JUSTICE` | Police, arrest, charges, fines, prosecution, defense, victims' rights, criminal procedure. |\n| `CONSUMER_DEBT` | Purchases, warranties, refunds, scams, debt collection, loans, bankruptcy, credit. |\n| `CONTRACTS` | Civil agreements; breach/interpretation not better captured by work/housing/consumer/business. |\n| `IMMIGRATION` | Visas, residence permits, asylum, citizenship, deportation, family migration. |\n| `BUSINESS` | Company formation, shareholders, commercial compliance, B2B disputes, self-employment. |\n| `DATA_PRIVACY` | Personal data, surveillance, GDPR, data deletion, consent, platform data practices. |\n| `INTELLECTUAL_PROPERTY` | Copyright, trademark, patent, trade secrets, licensing, infringement. |\n| `CIVIL_RIGHTS` | Discrimination outside work/housing, free speech, due process, equal treatment. |\n| `INTERNATIONAL_CROSS_BORDER` | Choice of law, jurisdiction, treaty-based questions, cross-border enforcement. |\n| `OTHER` | Genuinely legal but not covered above. |\n\nTwo selection rules carried into every classifier prompt:\n\n- Use `CONTRACTS` only when the issue is mainly about a civil agreement and is not better captured by `WORK`, `HOUSING`, `CONSUMER_DEBT`, or `BUSINESS`.\n- Use `INTERNATIONAL_CROSS_BORDER` only when the cross-border or jurisdictional aspect is central to the answer, not merely incidental.\n\n## Datasets\n\n| Hub ID | Contents |\n| ------ | -------- |\n| `AmirMohseni/WildChat-Legal-Classification-V2-Balanced` | Equal legal-guidance / non-guidance English rows (default) |\n| `AmirMohseni/WildChat-Legal-Classification-V2-LegalOnly` | `seeks_legal_guidance=true` rows only; includes extracted incidents |\n\nBoth are `DatasetDict` with splits `total`, `train`, `val`, and `test`. The balanced dataset uses a 90/5/5 partition; legal-only uses 80/10/10.\n\nRequired columns: `conversation` (list of `{role, content}`), `seeks_legal_guidance` (bool), `primary_topic` (string, empty for non-guidance rows). Additional v2 columns (`contains_legal_content`, `personalized_or_operational`, `topics`, `uncertainty`, `relevant_*_turn_ids`) are passed through but not consumed by the env's current scoring.\n\n## Install\n\n**From the Environments Hub** (after publishing):\n\n```bash\nuv run --with verifiers vf-install amirmohseni/legal-cls\n```\n\n**From this repository** (editable, for development):\n\n```bash\nuv pip install -e ./environments/legal_cls\nprime env install legal-cls -p ./environments\n```\n\n## Quickstart\n\n```bash\n# Balanced dataset, fast prompts, no abstention (default)\nprime eval run amirmohseni/legal-cls\n\n# Reasoning (chain-of-thought before answer)\nprime eval run amirmohseni/legal-cls -a '{\"prompt_style\": \"reasoning\"}'\n\n# Allow DONT_KNOW abstention (+1 correct / 0 abstain / -2 wrong by default; tune via wrong_penalty)\nprime eval run amirmohseni/legal-cls -a '{\"allow_dont_know\": true}'\n\n# Require the model to declare uncertainty (LOW / MEDIUM / HIGH) before its answer\nprime eval run amirmohseni/legal-cls -a '{\"require_uncertainty\": true}'\n\n# Legal-only dataset\nprime eval run amirmohseni/legal-cls \\\n  -a '{\"dataset_id\": \"AmirMohseni/WildChat-Legal-Classification-V2-LegalOnly\"}'\n```\n\n## Training\n\nFour configs in `configs/rl/` cover the 2 × 2 comparison of prompt style and abstention:\n\n| Config | `prompt_style` | `allow_dont_know` | Notes |\n| ------ | -------------- | ----------------- | ----- |\n| `legal-cls-fast.toml` | `fast` | `false` | Baseline — tags only, no CoT |\n| `legal-cls-fast-dk.toml` | `fast` | `true` | Baseline + abstention |\n| `legal-cls-reasoning.toml` | `reasoning` | `false` | CoT, no abstention |\n| `legal-cls-reasoning-dk.toml` | `reasoning` | `true` | CoT + abstention |\n\nLaunch a run:\n\n```bash\nprime rl run configs/rl/legal-cls-fast.toml\nprime rl run configs/rl/legal-cls-reasoning.toml\nprime rl run configs/rl/legal-cls-fast-dk.toml\nprime rl run configs/rl/legal-cls-reasoning-dk.toml\n```\n\nMonitor:\n\n```bash\nprime rl logs <run-id> -f\n```\n\n## Evaluation options\n\n| Flag | Example | Effect |\n| ---- | ------- | ------ |\n| `-m` | `-m openai/gpt-4.1` | Change model |\n| `-n` / `-r` | `-n 50 -r 3` | Examples / rollouts per example |\n| `prompt_style` | `\"reasoning\"` or `\"fast\"` | CoT vs tags-only prompt |\n| `allow_dont_know` | `true` / `false` | Enable abstention option |\n| `eval_split` | `\"val\"`, `\"test\"`, `\"total\"` | Which split to evaluate |\n\nExample — evaluate a trained adapter on the held-out test split:\n\n```bash\nprime eval run amirmohseni/legal-cls \\\n  -m openai/gpt-4.1 \\\n  -n 200 -r 1 \\\n  -a '{\"prompt_style\": \"reasoning\", \"eval_split\": \"test\"}'\n```\n\n## Environment arguments\n\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `dataset_id` | str | `AmirMohseni/WildChat-Legal-Classification-V2-Balanced` | Hub dataset ID |\n| `train_split` | str | `train` | Split used for training rollouts |\n| `eval_split` | str | `test` | Split used for evaluation |\n| `prompt_style` | str | `fast` | `reasoning` (XML `<reasoning>` + tags) or `fast` (tags only) |\n| `allow_dont_know` | bool | `false` | Allow `DONT_KNOW` in `<legal>` / `<topic>` |\n| `max_examples` | int | `-1` | Cap training examples after shuffle (`-1` = all) |\n| `eval_max_examples` | int | same as `max_examples` | Cap eval examples separately |\n| `seed` | int | `null` | Shuffle seed |\n| `rubric_legal_weight` | float | `1.0` | Weight for legal correctness in combined reward |\n| `rubric_topic_weight` | float | `1.0` | Weight for topic correctness in combined reward |\n| `secondary_topic_credit` | float | `0.0` | Credit awarded by `reward_topic` when the predicted topic is in the row's `topics` list but not the `primary_topic`. `0.0` (default) preserves legacy \"primary-only\" scoring; e.g. `0.5` gives partial credit for picking a co-labelled topic like `CONTRACTS` when the primary is `CONSUMER_DEBT`. |\n| `wrong_penalty` | float | `-2.0` | Reward returned by `reward_legal` / `reward_topic` for an in-domain wrong answer when `allow_dont_know = true` (ignored otherwise). Set more negative to push the policy toward `DONT_KNOW` on uncertain rows; abstaining beats guessing whenever `p_correct < \\|wrong_penalty\\| · p_wrong`. |\n| `require_uncertainty` | bool | `false` | Ask the model to emit `<uncertainty>LOW\\|MEDIUM\\|HIGH</uncertainty>` before `<legal>`. Purely diagnostic — does not change `reward_legal` or `reward_topic`. Adds three weight-0 metrics (`metric_uncertainty_low/medium/high`). |\n\n## Topic credit (`secondary_topic_credit`)\n\nV2 dataset rows carry both a single `primary_topic` and a `topics` list (1–3 reasonable labels). With `secondary_topic_credit=0.0` (default), `reward_topic` only rewards the exact `primary_topic` match, so picking a co-labelled topic is treated as wrong. Setting `secondary_topic_credit > 0` interpolates between the +1 (primary) and the wrong-answer floor:\n\n| Predicted topic vs gold | `secondary_topic_credit=0.0` (default) | e.g. `0.5` |\n| --- | --- | --- |\n| `pred == primary_topic` | +1.0 | +1.0 |\n| `pred ∈ topics`, not primary | `wrong_penalty` (with DK) / 0.0 (without) | **+0.5** |\n| `pred ∈ taxonomy`, not in `topics` | `wrong_penalty` (with DK) / 0.0 (without) | unchanged |\n| `pred == DONT_KNOW` | 0.0 | 0.0 |\n\nWhen `allow_dont_know = true`, `DONT_KNOW` is the only \"free\" escape. Truncated rollouts (no `<legal>` or `<topic>` tag emitted), hedged answers (multiple `<legal>` or `<topic>` tags), off-list values, and unparseable XML all incur `wrong_penalty`. `reward_legal` owns format checks for `<legal>` and `<uncertainty>`; `reward_topic` owns format checks for `<topic>` — this split avoids double-counting the same violation through the weighted sum in `reward_cls`.\n\nEnable from the CLI: `-a '{\"secondary_topic_credit\": 0.5}'`. Or in an RL config:\n\n```toml\n[[env]]\nid   = \"amirmohseni/legal-cls\"\nargs = { prompt_style = \"reasoning\", allow_dont_know = true, secondary_topic_credit = 0.5 }\n```\n\n## Metrics\n\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` | Weighted sum of `reward_legal` and `reward_topic` (training signal) |\n| `reward_legal` | `+1` correct, `0` abstain, `wrong_penalty` (default `-2.0`) wrong when `allow_dont_know`; else `+1`/`0` |\n| `reward_topic` | `+1` primary, `secondary_topic_credit` if in `topics`, `0` abstain, `wrong_penalty` (default `-2.0`) wrong when `allow_dont_know`; else `+1`/`0` |\n| `format_reward_func` | XML format adherence (diagnostic, weight 0) |\n| `metric_abstain_legal` | Abstention rate for the legal prediction (diagnostic, weight 0) |\n| `metric_abstain_topic` | Abstention rate for the topic prediction (diagnostic, weight 0) |\n| `metric_in_topics` | Fraction of legal-row rollouts whose predicted topic is in the gold `topics` list (primary or secondary). Diagnostic, weight 0. |\n| `metric_uncertainty_low` / `medium` / `high` | Distribution of declared `<uncertainty>` values. Only registered when `require_uncertainty = true`. Diagnostic, weight 0. Sum < 1 ⇒ rate of missing or off-list uncertainty values. |\n","encoding":"utf-8","truncated":false,"total_bytes":9782},"status":null}