{"data":{"kind":"file","path":"README.md","version_id":"nmj1hg0osr4be1eyc1ew2yub","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":1780,"modified_at":"2026-02-20T23:11:44.218000","content_hash":"c61ac6e400f133af9c921dcafe61742cb151bc684f9830d087c90d71ac7e84bb"},"entries":[],"content":"# sctpublic\n\nEvaluation environment for SCT-Bench public dataset.\n\n## Overview\n- **Environment ID**: `sctpublic`\n- **Short description**: Single-turn SCT dataset environment\n- **Tags**: medical, clinical, single-turn, eval\n\n## Datasets\n- **Primary dataset(s)**: SCT-Bench public \n- **Source links**: https://github.com/SCT-Bench/sctpublic\n- **Split sizes**: Evaluation only\n\n## Task\n- **Type**: Single-turn clinical reasoning evaluation\n- **Rubric overview**: Custom `sct_rubric` that normalizes the answer distribution so that the greatest score is always 1\n\n## Environment Arguments\n\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `reason` | bool | `False` | If True, prompts include an explanation requirement |\n| `few_shot` | bool | `False` | If True, includes 5 example ratings in the prompt |\n\n## Quickstart\nRun an evaluation with default settings:\n\n```bash\nprime eval run sctpublic -m \"openai/gpt-5-mini\" -n 5 -s\n```\n\n## Usage\nTo run an evaluation using `medarc-eval` with few-shot prompting and reasoning enabled:\n\n```bash\nmedarc-eval sctpublic -m \"openai/gpt-5-mini\" -n 5 -s --reason --few-shot\n```\n\n## Authors\nThis environment has been put together by:\n\nRatna Sagari Grandhi - ([@sagarigrandhi](https://github.com/sagarigrandhi))\n\n## Credits \nDataset:\n```bibtex\n@article{mccoy2025assessment,\n  title={Assessment of large language models in clinical reasoning: a novel benchmarking study},\n  author={McCoy, Liam G and Swamy, Rajiv and Sagar, Nidhish and Wang, Minjia and Bacchi, Stephen and Fong, Jie Ming Nigel and Tan, Nigel CK and Tan, Kevin and Buckley, Thomas A and Brodeur, Peter and others},\n  journal={NEJM AI},\n  volume={2},\n  number={10},\n  pages={AIdbp2500120},\n  year={2025},\n  publisher={Massachusetts Medical Society}\n}\n```\n","encoding":"utf-8","truncated":false,"total_bytes":1780},"status":null}