{"data":{"kind":"file","path":"README.md","version_id":"jtvyif4m2zn9e5mi06za5l43","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":1185,"modified_at":"2025-12-11T08:58:11.503000","content_hash":"6c2436c6692fea16fe67cb543ea185361af0356fa52afec3a03f9f048a120fe1"},"entries":[],"content":"# med-lm-eval\nAutomated LLM evaluation suite for medical tasks\n# MedQA Eval\n\nThis repository provides an evaluation environment for the [MedQA](https://huggingface.co/datasets/GBaker/MedQA-USMLE-4-options).\n\n## Usage\n\nTo run an evaluation using [vf-eval](https://github.com/PrimeIntellect-ai/verifiers) with the OpenAI API, use:\n\n```sh\nexport OPENAI_API_KEY=sk-...\nvf-eval medqa -m gpt-4.1-mini -n 5 -s\n```\n\nReplace `OPENAI_API_KEY` with your actual API key.\n\n## Environment\n\nThe evaluation environment is defined in `medqa.py` and uses the HuggingFace `GBaker/MedQA-USMLE-4-options` dataset.\n\n## Credits\n\nDataset:\n\n```bibtex\n@misc{jin2020diseasedoespatienthave,\n      title={What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams}, \n      author={Di Jin and Eileen Pan and Nassim Oufattole and Wei-Hung Weng and Hanyi Fang and Peter Szolovits},\n      year={2020},\n      eprint={2009.13081},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https://arxiv.org/abs/2009.13081}, \n}\n```\n\n### Authors\nThis environment has been put together by:\n\nAhmed Essouaied - ([@ahmedessouaied](https://github.com/ahmedessouaied))","encoding":"utf-8","truncated":false,"total_bytes":1185},"status":null}