{"data":{"kind":"file","path":"README.md","version_id":"sbra790g4rgv1epz1orxhel3","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2254,"modified_at":"2025-09-09T23:27:26.246000","content_hash":"b84394ba28838071416004d1f9eaeb716e87a1f0eab0cb29b107a60f1898a91e"},"entries":[],"content":"# APP-fraud-detection\n\n### Overview\n- **Environment ID**: `APP-fraud-detection`\n- **Short description**: Fraud detection environment for APP (Application Payment Platform) scams using SMS spam classification and document analysis\n- **Tags**: fraud-detection, security, classification, sms-spam, train, eval\n\n### Datasets\n- **Primary dataset(s)**: \n  - **ucirvine/sms_spam** – 5,574 English SMS messages labeled ham/spam for fraud detection training\n- **Source links**: \n  - [Hugging Face Dataset](https://huggingface.co/datasets/ucirvine/sms_spam)\n  - [UCI Machine Learning Repository](https://archive.ics.uci.edu/dataset/228/sms%2Bspam%2Bcollection)\n- **Split sizes**: Dynamically created 80/20 train/eval split from the source data\n\n### Task\n- **Type**: single-turn classification with structured output\n- **Parser**: JSON-structured action parsing for fraud indicators\n- **Rubric overview**: \n  - Base classification accuracy (spam vs ham)\n  - Risk score calibration bonuses\n  - Reasoning quality assessment\n  - Precision/recall metrics for spam detection\n\n### Quickstart\nRun an evaluation with default settings:\n\n```bash\nuv run vf-eval APP-fraud-detection\n```\n\nConfigure model and sampling:\n\n```bash\nuv run vf-eval APP-fraud-detection   -m gpt-4.1-mini   -n 20 -r 3 -t 1024 -T 0.7   -a '{\"key\": \"value\"}'  # env-specific args as JSON\n```\n\nNotes:\n- Use `-a` / `--env-args` to pass environment-specific configuration as a JSON object.\n\n### Environment Arguments\n\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `max_examples` | int | `-1` | Limit on dataset size (use -1 for all) |\n| `sources` | list[str] | `[\"sms_spam\"]` | Data sources to include in environment |\n| `shuffle` | bool | `true` | Shuffle dataset items between episodes |\n| `single_step_episodes` | bool | `true` | Complete episode after single classification |\n| `max_steps` | int | `dataset_size` | Maximum steps per episode |\n\n### Metrics\n\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` | Main scalar reward (base accuracy + bonuses) |\n| `sms_accuracy` | Classification accuracy on SMS spam/ham |\n| `sms_precision` | Precision for spam detection |\n| `sms_recall` | Recall for spam detection |\n| `risk_score` | Average risk calibration score |\n\n","encoding":"utf-8","truncated":false,"total_bytes":2254},"status":null}