{"data":{"kind":"file","path":"README.md","version_id":"hjtps186x3qyh95eqklwv9hf","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":1825,"modified_at":"2025-12-04T02:10:39.877000","content_hash":"2d95305cadb605c2a22b3c04bceacb59fb75543eef0da4124007637ce21856fe"},"entries":[],"content":"# linear-api-bench\n\n### Overview\n- **Environment ID**: `linear-api-bench`\n- **Short description**: Multi-turn agent evaluation on Linear API tasks using Agent Diff replicas\n- **Tags**: linear, api, tool-use, eval,\n\n\n### Datasets\n- **Primary dataset(s)**: `hubertmarek/linear-bench` - 40 Linear API tasks\n- **Source links**: [HuggingFace](https://huggingface.co/datasets/hubertmarek/linear-bench)\n\n### Task\n- **Type**: multi-turn tool use\n- **Parser**: Default\n- **Rubric overview**: Task-specific verification logic\n\n### Quickstart\n\n[Get your API key](https://agentdiff.dev)\n\n```bash\nexport AGENTDIFF_API_KEY=\"ad_live_sk_...\"\n```\n\nRun evaluation:\n```bash\nuv run vf-eval linear-api-bench -m gpt-5-mini -n 40 -c 5\n```\n\n### Environment Arguments\n\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `dataset_name` | str | `hubertmarek/linear-bench` | HuggingFace dataset |\n| `agentdiff_api_key` | str | env var | AgentDiff API key |\n| `max_turns` | int | `20` | Max conversation turns |\n\n### Metrics\n\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` | 1.0 if all test assertions pass, 0.0 otherwise |\n| `eval_passed` | Whether all assertions passed (boolean) |\n| `eval_score` | Score dict: `{total, passed, percent}` |\n| `eval_details` | Detailed assertion results (if available) |\n| `eval_diff` | Diff information (if available) |\n| `eval_error` | Error message if evaluation failed |\n\n\nTo save detailed diffs(eval_diff), assertion details:\n```bash\nuv run vf-eval linear-api-bench -m gpt-5 -n 40 -s \\\n  -C \"eval_passed,eval_score,eval_details,eval_diff,eval_error\"\n```\n\n\n### More\n\n**Website:** [agentdiff.dev](https://agentdiff.dev)  | **Repo:** [Github](https://github.com/AgentDiff/diff-the-universe)  |  **Docs:** [ Evaluation Docs](https://agentdiff.mintlify.app/core-concepts/evaluations)\n\n","encoding":"utf-8","truncated":false,"total_bytes":1825},"status":null}