{"data":{"kind":"file","path":"README.md","version_id":"mqzejwbl88mlusjo0e71tqqx","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6384,"modified_at":"2026-08-02T13:02:17.531000","content_hash":"b99a127ef92be7543fc6bca46942dd1db86a8208d980877059c842aa079c19b9"},"entries":[],"content":"# Weight & Diabetes Medical QA Environment\n\nA verifiers environment that evaluates LLM reasoning on weight management, BMI calculation, diabetes risk assessment, and treatment guidelines.\n\n## Quick Start\n\n```bash\n# Install\nprime env install ashav3652/weight-diabetes-qa\n\n# Run evaluation\nprime eval run ashav3652/weight-diabetes-qa -m meta-llama/Llama-3.2-1B-Instruct\n\n# View results\nprime eval results --last\n```\n\n## Overview\n\nThis environment tests a model's ability to:\n\n- **Calculate BMI** and classify weight categories (WHO standards)\n- **Assess diabetes risk** using clinical markers (fasting glucose, HbA1c, OGTT)\n- **Explain pathophysiology** of obesity-diabetes connection (insulin resistance, inflammation)\n- **Recommend treatments** aligned with ADA/WHO guidelines (metformin, lifestyle, GLP-1 agonists)\n- **Manage complications** including micro/macrovascular diseases\n\n## Dataset\n\n25 expert-crafted medical QA pairs covering:\n\n| Topic | Questions |\n|-------|-----------|\n| BMI Calculations | 4 questions with numerical verification |\n| Diabetes Risk | 4 questions on risk factors and diagnosis |\n| Weight Management | 4 questions on strategies and mechanisms |\n| Complications | 3 questions on retinopathy, nephropathy, neuropathy |\n| Treatment | 3 questions on metformin, GLP-1 agonists, insulin |\n| Prevention | 2 questions on lifestyle modification |\n| Nutrition | 2 questions on glycemic index, Mediterranean diet |\n| Special Populations | 2 questions on gestational diabetes, pediatric |\n| Hormonal | 1 question on insulin/leptin/ghrelin regulation |\n\n## Rubric (4 metrics)\n\n| Metric | Weight | Description |\n|--------|--------|-------------|\n| Accuracy | 0.40 | Numerical and factual correctness with tolerance matching |\n| Medical Reasoning | 0.25 | Quality of clinical reasoning, use of medical terminology |\n| Guideline Compliance | 0.20 | Alignment with WHO/ADA standards and categories |\n| Completeness | 0.15 | Thoroughness of response, structured explanation |\n\n## Architecture\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│               weight-diabetes-qa v0.1.3                         │\n├─────────────────────────────────────────────────────────────────┤\n│                                                                 │\n│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐      │\n│  │  MEDICAL QA  │───▶│    MODEL     │───▶│  COMPLETION  │      │\n│  │  (25 items)  │    │  (LLM eval)  │    │  (response)  │      │\n│  └──────────────┘    └──────────────┘    └──────┬───────┘      │\n│                                                  │               │\n│                                                  ▼               │\n│  ┌─────────────────────────────────────────────────────────┐   │\n│  │                    RUBRIC (4 metrics)                     │   │\n│  │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐│   │\n│  │  │ Accuracy │  │ Medical  │  │Guideline │  │Complete- ││   │\n│  │  │  (0.40)  │  │Reasoning │  │Compliance│  │  ness    ││   │\n│  │  │          │  │  (0.25)  │  │  (0.20)  │  │  (0.15)  ││   │\n│  │  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘│   │\n│  │       │              │              │              │      │   │\n│  │       ▼              ▼              ▼              ▼      │   │\n│  │  ┌─────────────────────────────────────────────────────┐ │   │\n│  │  │              WEIGHTED COMPOSITE SCORE                │ │   │\n│  │  └─────────────────────────────────────────────────────┘ │   │\n│  └─────────────────────────────────────────────────────────┘   │\n│                                                                 │\n└─────────────────────────────────────────────────────────────────┘\n```\n\n## Installation\n\n```bash\n# From Prime Intellect Hub\nprime env install ashav3652/weight-diabetes-qa\n\n# From source\ngit clone <repo>\ncd weight-diabetes-env\nuv pip install -e .\n```\n\n## Running Tests\n\n```bash\ncd weight-diabetes-env\npytest tests/ -v\n```\n\n## Troubleshooting\n\n### Problem: Model always scores 0.0 on accuracy\n\n**Symptoms:** All accuracy scores are 0.0 despite seemingly correct answers.\n\n**Solutions:**\n1. Check if model output includes the numeric BMI value (e.g., \"27.8\")\n2. Ensure model includes the weight category keyword (e.g., \"overweight\")\n3. The accuracy function uses tolerance matching for numbers within 0.1\n\n### Problem: Low medical reasoning scores\n\n**Symptoms:** Medical reasoning score consistently below 0.3.\n\n**Solutions:**\n1. Model should use medical terminology (BMI, insulin, glucose, etc.)\n2. Include calculations with kg/m notation\n3. Add cautionary language (consult, doctor, varies)\n\n## Version History\n\n- **v0.1.3**: Added pytest tests, improved README with architecture diagram\n- **v0.1.2**: Added tests directory, fixed ThinkParser issue\n- **v0.1.1**: Removed ThinkParser, used plain Parser\n- **v0.1.0**: Initial release with 25 QA pairs\n\n## License\n\nMIT\n","encoding":"utf-8","truncated":false,"total_bytes":6384},"status":null}