{"data":{"kind":"file","path":"README.md","version_id":"owbqttb0j9vsk5rahds5ya5y","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2904,"modified_at":"2025-10-28T21:49:53.535000","content_hash":"f74b47eaf8e4b146267b9ded8c3ce65736060b6572c55825f067548d1ea7075e"},"entries":[],"content":"# flawedfictions\n\n### Overview\n- **Environment ID**: `flawedfictions`\n- **Short description**: Single-turn plot-hole detection task in short stories, proposed by Ahuja et al., 2025. Includes binary prediction task and line identification task.\n- **Tags**: single-turn, story-understanding\n\n### Datasets\n- **Primary dataset(s)**: `kahuja/flawed-fictions`, short or long splits.\n- **Source links**: https://huggingface.co/datasets/kahuja/flawed-fictions\n- **Split sizes**: By default 331/83 (80/20 train eval split on flawed_fictions subset).\n\n### Task\n- **Type**: single-turn\n- **Parser**: custom\n- **Rubric overview**: Binary reward for plot-hole detection (accuracy), optional line identification and format rewards \n\n### Quickstart\nRun an evaluation with default settings:\n\n```bash\nuv run vf-eval flawedfictions\n```\n\nConfigure model and sampling:\n\n```bash\nuv run vf-eval flawedfictions   -m gpt-4.1-mini   -n 20 -r 3 -t 1024 -T 0.7   -a '{\"key\": \"value\"}'  # env-specific args as JSON\n```\n\nNotes:\n- Use `-a` / `--env-args` to pass environment-specific configuration as a JSON object.\n\n### Environment Arguments\n\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `num_train_examples` | int | `-1` | Number of train examples (use -1 for all) |\n| `num_eval_examples` | int | `-1` | Number of eval examples (use -1 for all) |\n| `train_source` | str | `\"flawed_fictions\"` | HF split for training (`flawed_fictions` or `flawed_fictions_long`) |\n| `eval_source` | str | `\"flawed_fictions\"` | HF split for eval (`flawed_fictions` or `flawed_fictions_long`) |\n| `prompt_name` | str | `\"simple_boxed_prompt.txt\"` | Prompt format (`simple_boxed_prompt.txt`, `conterror_detector_prompt.txt`, `conterror_detector_prompt_cot.txt`) |\n| `split_ratio` | float | `0.8` | Train/eval split used only when train=eval source and counts are `-1` |\n| `binary_reward_weight` | float | `1.0` | Weight for binary correctness (error/no‑error) |\n| `binary_format_reward_weight` | float | `0.0` | Weight for binary format (e.g., boxed yes/no or paper tags) |\n| `line_identification_reward_weight` | float | `0.0` | Weight for localization correctness (lines overlap) |\n| `line_identification_format_reward_weight` | float | `0.0` | Weight for localization format (requires line tags for positives) |\n\n### Metrics\n\n| Metric | Meaning |\n| ------ | ------- |\n| `binary_format` | 1 if binary is correctly formatted (boxed yes/no or paper tags), else 0 |\n| `binary_accuracy` | 1 if parsed binary label equals `cont_error`, else 0 |\n| `localization_format` | 1 if (binary is formatted and, for positives, both line blocks present); negatives count as 1 |\n| `localization_full` | 1 if (binary is correct) and predicted error/contradicted lines overlap GT after sentence mapping, else 0 |\n| `reward` | Weighted sum: `wf*binary_format + wb*binary_accuracy + wl_f*localization_format + wl*localization_full` |\n","encoding":"utf-8","truncated":false,"total_bytes":2904},"status":null}