{"data":{"kind":"file","path":"README.md","version_id":"k7x5ser8kitnl8pgn9j4nte2","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":4475,"modified_at":"2025-10-09T17:44:42.445000","content_hash":"361759d12c14f0ab62f7719a796b6fa29c0bee33ac7ebd8d2ef8d80f6850fa8f"},"entries":[],"content":"# AgentClinic Environment\n\nMulti-agent medical diagnosis environment for evaluating LLMs on clinical diagnosis through interactive conversations.\n\n## Quick Start\n\n### 1. Install\n\n```bash\nuv pip install -e .\n```\n\n### 2. Set API Keys\n\n```bash\n# Required for helper agents (patient, measurement, moderator)\nexport OPENAI_API_KEY=\"your-key\"\n\n# Optional: For using other models\nexport ANTHROPIC_API_KEY=\"your-key\"\nexport MISTRAL_API_KEY=\"your-key\"\n\n```\n\n---\n\n## Running Evaluations\n\n### Basic Command Structure\n\n```bash\nuv run --active -m verifiers.scripts.eval \\\n  -m MODEL_NAME \\                    # Doctor model (evaluated)\n  -b API_BASE_URL \\                  # Doctor API endpoint\n  -k API_KEY_VAR \\                   # Doctor API key variable name\n  agentclinic \\\n  -n NUM_CASES \\                     # Number of cases to evaluate\n  --rollouts-per-example 3 \\         # Rollouts \n  --max-concurrent 2 \\               # Parallel requests\n  -T 0.0 \\                          # Temperature\n  -s \\                              # Save results\n  --env-args '{\n    \"dataset_path\": \"DATASET.jsonl\",\n    \"patient_model\": \"MODEL\",\n    \"measurement_model\": \"MODEL\",\n    \"moderator_model\": \"MODEL\"\n  }'\n```\n\n---\n\n## Examples\n\n### MedQA with GPT-4o-mini (all agents)\n\n```bash\nexport OPENAI_API_KEY=\"your-key\"\n\nuv run --active -m verifiers.scripts.eval \\\n  -m gpt-4o-mini \\\n  -b https://api.openai.com/v1 \\\n  -k OPENAI_API_KEY \\\n  agentclinic \\\n  -n 50 \\\n  --rollouts-per-example 3 \\\n  --max-concurrent 2 \\\n  -T 0.0 \\\n  -s \\\n  --env-args '{\"dataset_path\": \"agentclinic_medqa_extended.jsonl\"}'\n```\n\n### NEJM with GPT-4o-mini (all agents)\n\n```bash\nexport OPENAI_API_KEY=\"your-key\"\n\nuv run --active -m verifiers.scripts.eval \\\n  -m gpt-4o-mini \\\n  -b https://api.openai.com/v1 \\\n  -k OPENAI_API_KEY \\\n  agentclinic \\\n  -n 50 \\\n  --rollouts-per-example 3 \\\n  --max-concurrent 2 \\\n  -T 0.0 \\\n  -s \\\n  --env-args '{\"dataset_path\": \"agentclinic_nejm_extended.jsonl\"}'\n```\n\n### Mistral Large (all agents)\n\n```bash\nexport MISTRAL_API_KEY=\"your-key\"\n\nuv run --active -m verifiers.scripts.eval \\\n  -m mistral-large-latest \\\n  -b https://api.mistral.ai/v1 \\\n  -k MISTRAL_API_KEY \\\n  agentclinic \\\n  -n 50 \\\n  --rollouts-per-example 3 \\\n  --max-concurrent 2 \\\n  -T 0.0 \\\n  -s \\\n  --env-args '{\n    \"dataset_path\": \"agentclinic_nejm_extended.jsonl\",\n    \"patient_model\": \"mistral-large-latest\",\n    \"patient_backend\": \"mistral\",\n    \"measurement_model\": \"mistral-large-latest\",\n    \"measurement_backend\": \"mistral\",\n    \"moderator_model\": \"mistral-large-latest\",\n    \"moderator_backend\": \"mistral\"\n  }'\n```\n\n### Claude 3.5 Sonnet (doctor) + GPT-4o-mini (helpers)\n\n```bash\nexport ANTHROPIC_API_KEY=\"your-key\"\nexport OPENAI_API_KEY=\"your-key\"\n\nuv run --active -m verifiers.scripts.eval \\\n  -m claude-3-5-sonnet-20241022 \\\n  -b https://api.anthropic.com/v1 \\\n  -k ANTHROPIC_API_KEY \\\n  agentclinic \\\n  -n 50 \\\n  --rollouts-per-example 3 \\\n  --max-concurrent 2 \\\n  -T 0.0 \\\n  -s \\\n  --env-args '{\"dataset_path\": \"agentclinic_medqa_extended.jsonl\"}'\n```\n\n\n\n---\n\n## Configuration Options\n\n### Agent Configuration\n\nConfigure each agent separately via `--env-args`:\n\n```json\n{\n  \"dataset_path\": \"agentclinic_medqa_extended.jsonl\",\n\n  // Patient Agent\n  \"patient_model\": \"gpt-4o-mini\",\n  \"patient_backend\": \"auto\",\n  \"patient_api_key\": null,\n  \"patient_api_base\": null,\n\n  // Measurement Agent\n  \"measurement_model\": \"gpt-4o-mini\",\n  \"measurement_backend\": \"auto\",\n  \"measurement_api_key\": null,\n  \"measurement_api_base\": null,\n\n  // Moderator/Judge Agent\n  \"moderator_model\": \"gpt-4o-mini\",\n  \"moderator_backend\": \"auto\",\n  \"moderator_api_key\": null,\n  \"moderator_api_base\": null,\n\n  // Other options\n  \"max_turns\": 20,\n  \"use_think\": false\n}\n```\n\n### Supported Backends\n\n- `openai` - OpenAI models (gpt-4, gpt-4o-mini, etc.)\n- `anthropic` - Anthropic models (claude-3-5-sonnet, etc.)\n- `mistral` - Mistral models (mistral-large-latest, etc.)\n- `gemini` - Google Gemini models\n- `vllm` - Local vLLM server\n- `auto` - Auto-detect from model name (default)\n\n### Datasets\n\n- **MedQA Extended** (214 cases): `agentclinic_medqa_extended.jsonl`\n- **NEJM Extended** (120 cases): `agentclinic_nejm_extended.jsonl`\n\n---\n\n## Agent Roles\n\n- **Doctor** (evaluated model): Asks questions, orders tests, makes diagnosis\n- **Patient** (helper): Simulates patient responses based on case data\n- **Measurement** (helper): Returns test results from case data\n- **Moderator** (helper): Judges if diagnosis matches ground truth\n\n","encoding":"utf-8","truncated":false,"total_bytes":4475},"status":null}