{"data":{"kind":"file","path":"README.md","version_id":"lpcxfehcxwcq4esoccsps62s","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":1682,"modified_at":"2026-02-20T23:11:44.201000","content_hash":"f63dc615cf1045645c0074b71aed497d58ede842092c2a426d5f92e58294436f"},"entries":[],"content":"# medcasereasoning\n\n## Overview\n- **Environment ID**: `medcasereasoning`\n- **Short description**: MedCaseReasoning dataset from Stanford (Wu et al. 2025) — open-ended clinical case diagnosis evaluated by LLM judge\n- **Tags**: medical, clinical, single-turn, diagnosis, llm-judge, train, eval\n\n## Datasets\n- **Primary dataset(s)**: MedCaseReasoning — clinical case presentations with ground truth final diagnoses\n- **Source links**: [zou-lab/MedCaseReasoning](https://huggingface.co/datasets/zou-lab/MedCaseReasoning)\n- **Split sizes**: 13.1k (train) / 500 (val) / 897 (test)\n\n## Task\n- **Type**: single-turn\n- **Rubric overview**: LLM-as-a-Judge — judge is asked \"Is our predicted diagnosis correct (yes/no)?\"; returns 1.0 if the judge answers \"yes\", else 0.0\n\n## Quickstart\nRun an evaluation with default settings:\n\n```bash\nprime eval run medcasereasoning -m \"openai/gpt-5-mini\" -n 5 -s\n```\n\nConfigure model and sampling:\n\n```bash\nmedarc-eval medcasereasoning -m \"openai/gpt-5-mini\" -n 20 -s\n```\n\nJudge example:\n\n```bash\nmedarc-eval medcasereasoning -m \"openai/gpt-5-mini\" -n 20 -s --judge-model \"openai/gpt-5-nano\"\n```\n\n## Environment Arguments\n\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `judge_model` | str | `\"openai/gpt-5-nano\"` | Model to use for LLM-as-a-Judge evaluation |\n| `judge_base_url` | str \\| None | `None` | Optional base URL for judge model API |\n| `judge_api_key` | str \\| None | `None` | Optional API key for judge model (defaults to `OPENAI_API_KEY`) |\n\n## Metrics\n\n| Metric | Meaning |\n| ------ | ------- |\n| `medical_diagnosis_reward_func` | (weight 1.0): 1.0 if judge confirms the diagnosis is correct, else 0.0 |\n","encoding":"utf-8","truncated":false,"total_bytes":1682},"status":null}