{"data":{"kind":"file","path":"README.md","version_id":"sq3ds1q8io08j36m0m8yntqe","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5909,"modified_at":"2026-07-07T20:56:31.493000","content_hash":"3e910010ae2584d0c111d7c780a12b156792030ae2b1768f4ce4e277765ebb06"},"entries":[],"content":"# set-diagnosis-tool-call\n\n### Overview\n- **Environment ID**: `set-diagnosis-tool-call`\n- **Short description**: Environment for evaluating **clinical diagnosis assignment via tool calls**. Models must analyze patient case reports and generate correct JSON tool calls to record the appropriate ICD-10 diagnosis with an acute/chronic status.\n- **Tags**: `tool-calling`, `medical`, `function-calling`, `icd-10`\n\n### Datasets\n- **Primary dataset(s)**: `mkurman/clinical-case-icd10-diagnosis`\n- **Source links**: https://huggingface.co/datasets/mkurman/clinical-case-icd10-diagnosis\n- **Split sizes**: single `train` split (1,798 cases) split 90/10 into ~1,618 train / ~180 test (seed 42)\n- Each case is a de-identified clinical case report (`input`) with **one ground-truth\n  primary diagnosis**: an ICD-10-CM `icd10_code`, a human-readable `diagnosis` name,\n  and a `status` of `ACUTE` or `CHRONIC`.\n\n### Task\n- **Type**: multi-turn\n- **Parser**: custom (extracts JSON from `<tool_call>` tags and native OAI `tool_calls`)\n- **Rubric overview**: Scores diagnosis coverage and precision across all turns. Models must output a tool call in the format:\n\n```xml\n<tool_call>\n{\n  \"name\": \"set_diagnosis\",\n  \"arguments\": {\n    \"icd10_code\": \"E05.01\",\n    \"description\": \"Thyrotoxicosis with diffuse goiter with thyrotoxic crisis or storm\",\n    \"status\": \"acute\"\n  }\n}\n</tool_call>\n```\n\n### How this differs from `order-medical-test-tool-call`\nThis environment is a port of the medical-test-ordering environment, retargeted\nfrom ordering tests to recording diagnoses:\n\n| order-medical-test-tool-call | set-diagnosis-tool-call |\n| ---------------------------- | ----------------------- |\n| `order_medical_test`         | `set_diagnosis`         |\n| `test_code` (identity)       | `icd10_code` (identity) |\n| `priority` (near-miss field) | `status` (near-miss field) |\n| `search_medical_tests`       | `search_diagnosis_codes` |\n| `list_medical_tests`         | `list_diagnosis_codes`  |\n| `finish_ordering_tests`      | `finish_setting_diagnoses` |\n| local test-code catalog (~1.5k) | real ICD-10-CM catalog (~98k, via `simple-icd-10-cm`) |\n\n- **Identity** = `icd10_code`. A correct code with the wrong `status` is a\n  **near-miss** (partial credit), not a wrong diagnosis.\n- **`status`** is `acute` or `chronic`. Field matching is **case-insensitive**\n  (`ACUTE` matches `acute`, `i10` matches `I10`), so casing is never a silent\n  reward killer.\n- **`description`** is required by the tool and echoed back, but is **not** part of\n  reward matching — it is free text with many valid phrasings.\n- **Code validity** is lenient: a code is accepted if it is in the ICD-10-CM catalog\n  **or** is a well-formed ICD-10 code (regex `^[A-Z][0-9][0-9A-Z](\\.[0-9A-Z]{1,4})?$`).\n  This avoids rejecting legitimate answers from other ICD-10 editions (e.g. WHO\n  ICD-10 `I43.1`, which ICD-10-CM writes as bare `I43`). All ground-truth codes in\n  the dataset are members of the ICD-10-CM catalog.\n\n### Installation\n\n```bash\nprime install set-diagnosis-tool-call\n```\n\n### Quickstart\n\nRun an evaluation with default settings:\n\n```bash\nprime eval set-diagnosis-tool-call\n```\n\nWith a local model:\n\n```bash\nexport DUMMY_KEY=\"dummy\"\nprime eval set-diagnosis-tool-call \\\n  -m zai-org/GLM-4.7-Flash \\\n  -b http://localhost:8001/v1 \\\n  -k DUMMY_KEY \\\n  -n 10 -r 1 -t 8192 -c 1\n```\n\nConfigure model and sampling:\n\n```bash\nprime eval set-diagnosis-tool-call \\\n  -m gpt-4.1-mini \\\n  -n 20 -r 3 -t 1024 -T 0.7\n```\n\n### Metrics\n\nThe rubric scores multiple dimensions with weighted contributions:\n\n| Metric | Weight | Meaning |\n| ------ | ------ | ------- |\n| `coverage_score` | 0.45 | Fraction of ground-truth diagnoses correctly recorded (exact) |\n| `precision_score` | 0.15 | Fraction of model's diagnoses that are correct (exact) |\n| `code_identity_coverage_score` | 0.20 | Credit for correct ICD-10 code despite wrong status |\n| `code_identity_precision_score` | 0.10 | Distinguish near-miss from wrong code |\n| `json_valid_score` | 0.05 | At least one valid `set_diagnosis` tool call |\n| `num_unique_diagnoses_score` | 0.05 | Encourages recording diverse diagnoses |\n| `reward` | 1.00 | Main scalar reward (weighted sum of above) |\n\n> **⚠️ Training note (GRPO / prime-rl):** The sub-scores must have **non-zero weights** for RL training. A pure binary coverage reward causes within-group rewards to collapse, GRPO advantage to go to zero, and prime-rl's `zero_advantage` filter to drop the batch. The partial-credit weights above create within-group reward variance so advantages are non-zero and the trainer receives gradient signal. For **eval-only** runs, a binary coverage-only weighting is fine.\n\n### Anti-hacking\n\nDiagnoses are recorded one per turn. Duplicate diagnoses (same `icd10_code` +\n`status`) are rejected and do not add credit, so repeatedly recording a single\ngeneric code scores no better than recording it once. Coverage counts unique\ncorrect matches against the ground truth. Since precision penalizes extra wrong\ndiagnoses, spraying many codes also scores worse than recording the correct one.\n\n### Tools\n\nThe model is provided with four tools:\n\n- `search_diagnosis_codes(query, limit)` — search the ICD-10 catalog by clinical name or partial code (does not record a diagnosis)\n- `list_diagnosis_codes(offset, limit)` — browse the ICD-10 catalog by page (does not record a diagnosis)\n- `set_diagnosis(icd10_code, description, status)` — record a diagnosis (`status` is `acute` or `chronic`)\n- `finish_setting_diagnoses(reason)` — end the rollout (plain-text `DONE` also accepted)\n\n### Example\n\n**Input**: A de-identified clinical case report describing a patient presentation.\n\n**Expected Output**:\n```xml\n<tool_call>\n{\n  \"name\": \"set_diagnosis\",\n  \"arguments\": {\n    \"icd10_code\": \"I26.09\",\n    \"description\": \"Other pulmonary embolism with acute cor pulmonale\",\n    \"status\": \"acute\"\n  }\n}\n</tool_call>\n```\n","encoding":"utf-8","truncated":false,"total_bytes":5909},"status":null}