{"data":{"kind":"file","path":"README.md","version_id":"r99g742h9cf8zzi8p2l7l3ca","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":7290,"modified_at":"2026-07-12T13:52:06.196000","content_hash":"bdba2d519e1a08a6a36d115f25e3cae02dd6ca693f3d89382f3d8c760a20700d"},"entries":[],"content":"# MedFact-Bench\n\nMedFact-Bench is a zero-shot environment for biomedical claim verification and\nevidence attribution. It implements the protocol introduced in\n[Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attribution](https://arxiv.org/abs/2603.05308)\nusing the native Verifiers 0.2 Taskset/Harness API.\n\n- **Environment ID:** `medfact-bench`\n- **Task:** single-turn, three-class medical claim verification\n- **Tags:** medical, fact-verification, evidence-attribution, evaluation\n- **Maintainer:** Kouate Muhamed ([KouateMuhamed](https://github.com/KouateMuhamed))\n\n## Dataset\n\n- **Dataset:** [`ncbi/MedFact-Bench`](https://huggingface.co/datasets/ncbi/MedFact-Bench)\n- **Pinned revision:** `249028caf7ad5a3e63331269a606f4b2696693ed`\n\n| Component | Examples |\n| --- | ---: |\n| SciFact | 340 |\n| HealthVer | 903 |\n| MedAESQA | 9,106 |\n| PubMedQA-Fact | 500 |\n| BioASQ-Fact | 3,425 |\n| **Total** | **14,274** |\n\nThe loader preserves row order and duplicate rows. It validates all required fields,\ncomponent names, labels, and the shared system prompt. A local Parquet file can be\nused instead of Hugging Face by setting `taskset.dataset_path`.\n\nMedFact-Bench is exposed only as an evaluation split. Training requests fail rather\nthan silently using held-out benchmark examples.\n\n## Protocol\n\nEach example contains a source article and a medical claim. The model returns a\nfive-point evidence-attribution score, which maps to the benchmark labels as follows:\n\n| Score | Prediction |\n| ---: | --- |\n| `1`, `2` | `SUPPORT` |\n| `0` | `NEI` |\n| `-1`, `-2` | `CONTRADICT` |\n\nThe parser reads the first `<score>...</score>` occurrence and accepts only an integer\nfrom `-2` through `2`. Missing, malformed, and out-of-range scores are classified as\n`INVALID`, receive zero accuracy, and remain in all reporting denominators.\n\n### System Prompt\n\nThe dataset-provided system prompt is passed to the model unchanged:\n\n```text\nYou are a fact-checking expert trained in evidence-based medicine. Your task is to evaluate how strongly an *article* agrees or disagrees with a *claim*. The *article* is retrieved from a search engine using the *claim* as the query.\n\nUse the following five-point scale:\n   - **-2 Strong Contradiction**  – The article clearly and directly refutes the claim.\n   - **-1 Partial Contradiction** – The article provides mixed or indirect evidence against the claim.\n   - ** 0 Neutral / Unrelated**   – The article does not address the claim, offers insufficient information, or is irrelevant to the claim.\n   - ** 1 Partial Agreement**\t – The article offers some indirect or tentative support for the claim.\n   - ** 2 Strong Agreement**\t – The article explicitly and strongly supports the claim.\n\nNote that the *article* might not describe the exact same subjects, interventions, or measurements as the *claim*. In this case, please note the difference and assign a score of 0.\n\nOutput in two parts only and do not output anything else:\n<think>[your detailed, step‐by‐step explanation for scoring]</think>\n<score>[the integer score only, i.e., -2, -1, 0, 1, or 2]</score>\n```\n\nThe square brackets identify placeholder content. They are not required in model output.\n\n### User Prompt\n\n```text\nArticle:\n{article}\n\nClaim:\n{claim}\n```\n\n## Installation\n\nInstall the local checkout:\n\n```bash\nuv sync\n```\n\nInside a Prime Lab workspace, clone the repository into the underscore-normalized\nenvironment directory before using the local environment installer:\n\n```bash\ngit clone https://github.com/KouateMuhamed/medfact-bench.git environments/medfact_bench\nprime env install medfact-bench --path environments --prerelease\n```\n\nAfter the first public release, install the immutable Hub version:\n\n```bash\nprime env install KouateMuhamed/medfact-bench@0.1.0 --prerelease\n```\n\n## Evaluation\n\nRun the reproducible zero-shot configuration:\n\n```bash\nuv run eval @ configs/eval/zero-shot.toml\n```\n\nRun a small evaluation through the Prime CLI:\n\n```bash\nprime eval run medfact-bench \\\n  -m \"openai/gpt-5-mini\" \\\n  -n 5 \\\n  -r 1\n```\n\nVerifiers 0.2 configuration fields are namespaced under the taskset:\n\n```toml\n[taskset]\nid = \"medfact-bench\"\nsubset = \"scifact\"\nsplit = \"eval\"\ndataset_path = \"/absolute/path/to/MedFact-Bench.parquet\"\n\n[harness]\nid = \"medfact-bench\"\n```\n\nInference model, endpoint, concurrency, and hardware are caller-owned. Sampling for\npaper comparisons should use one rollout, temperature `0`, and at most 1,024 output tokens.\n\n## Metrics\n\n| Metric | Role | Meaning |\n| --- | --- | --- |\n| `accuracy` | reward, weight 1.0 | Exact three-class accuracy |\n| `parseable_score` | metric | Whether a valid five-point score was parsed |\n| `strict_format` | metric | Exact compliance with one think block followed by one score block |\n\nThe average reward is example-weighted micro accuracy. The paper's primary metric is\nthe unweighted mean of the five component accuracies.\n\n## Reporting\n\nVerifiers 0.2 writes `traces.jsonl`. Generate the benchmark report with either command:\n\n```bash\nmedfact-report outputs/path/to/traces.jsonl\npython -m medfact_bench.reporting outputs/path/to/traces.jsonl\n```\n\nThe reporter also accepts legacy `results.jsonl`. For legacy runs it updates a sibling\n`metadata.json` non-destructively when present. Use `--no-update-metadata` to disable\nthat behavior.\n\nThe versioned report includes parse and format rates, micro and macro accuracy,\nper-component accuracy, per-class precision/recall/F1, macro F1, a confusion matrix\nincluding `INVALID`, and coverage diagnostics. `paper_macro_accuracy` is populated only\nwhen all 14,274 examples have exact component coverage.\n\n## Published Results\n\nFigure 3 of the Med-V1 paper reports the following zero-shot accuracies.\n\n### Small Models\n\n| Model | SciFact | HealthVer | MedAESQA | PubMedQA-Fact | BioASQ-Fact | Macro |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: |\n| Llama-3.2-3B-Instruct | .600 | .435 | .437 | .550 | .531 | .511 |\n| Qwen2.5-3B-Instruct | .638 | .447 | .577 | .470 | .444 | .515 |\n\n### Frontier Models\n\n| Model | SciFact | HealthVer | MedAESQA | PubMedQA-Fact | BioASQ-Fact | Macro |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: |\n| Llama-3.3-70B-Instruct | .812 | .597 | .687 | .774 | .771 | .728 |\n| o3-mini | .832 | .591 | .733 | .778 | .747 | .736 |\n| GPT-4o-mini | .847 | .580 | .704 | .734 | .720 | .717 |\n| GPT-4o | .832 | .576 | .717 | .776 | .777 | .736 |\n| GPT-5 | .818 | .615 | .703 | .788 | .753 | .735 |\n\n### Fine-Tuned Models\n\n| Model | SciFact | HealthVer | MedAESQA | PubMedQA-Fact | BioASQ-Fact | Macro |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: |\n| Med-V1-L3B | .844 | .575 | .748 | .746 | .725 | .728 |\n| Med-V1-Q3B | .856 | .588 | .733 | .764 | .717 | .732 |\n\nMacro is the unweighted mean of the five component accuracies. Evaluation manifests\nshould preserve dataset, model, dependency, prompt, and sampling revisions.\n\n## Citation\n\nIf this environment supports research, cite both this software and the Med-V1 paper.\nCitation metadata is available in [`CITATION.cff`](CITATION.cff).\n\n## Author\n\nKouate Muhamed  \nEmail: [muhamed.kouate@icloud.com](mailto:muhamed.kouate@icloud.com)\n\n## License\n\nThis environment is released under the MIT License. Dataset and upstream attribution\nare documented in [`NOTICE`](NOTICE).\n","encoding":"utf-8","truncated":false,"total_bytes":7290},"status":null}