{"data":{"kind":"file","path":"README.md","version_id":"vhf2glyzkfuxqss90etu6a0m","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3619,"modified_at":"2026-08-31T12:05:55.125000","content_hash":"40e84fec31bd77441c4faaf2516c005b1a04b1daf02cfc94da8ad24b98ba790a"},"entries":[],"content":"# mcp-sqlite\n\n### Overview\n- **Environment ID**: `mcp-sqlite`\n- **Short description**: Multi-turn tool-use environment where a model answers a question\n  about a synthetic, intentionally messy SQLite database through a 3-tool read-only MCP\n  server, graded by a reference judge against ground truth resolved from the database at\n  load time.\n- **Tags**: sql, sqlite, mcp, tool-use, database, agentic, multi-turn, llm-judge,\n  schema-reasoning\n\n### Datasets\n- **Primary dataset(s)**: a synthetic, deterministically generated SQLite database\n  (`mcp_sqlite/data/parts.sqlite`) with 57 hand-written tasks and their reference SQL\n- **Source links**: [generate_db.py](mcp_sqlite/scripts/generate_db.py) (database),\n  [tasks.py](mcp_sqlite/tasks.py) (tasks)\n- **Split sizes**: 57 tasks — `ambiguity` 12, `multi_hop` 11, `temporal` 10,\n  `schema_discovery` 9, `unanswerable` 8, `single_table` 7\n\n### Task\n- **Type**: multi-turn tool use (MCP)\n- **Parser**: none — the model's final chat message is graded directly\n  (`view: \"last_reply\"`)\n- **Rubric overview**: a `ReferenceJudge` grades the final reply correct/incorrect against\n  resolved ground truth (weight 1.0); schema-coverage and free-form diagnostics are logged\n  at weight 0.0 — see Metrics\n\n### Quickstart\nRun an evaluation with default settings:\n\n```bash\nuv run eval mcp-sqlite\n```\n\nConfigure model, sample size, and the agent loop:\n\n```bash\nuv run eval mcp-sqlite -n 5 --model gpt-4.1-mini \\\n    --env.agent.max-turns 20 --env.agent.runtime.type subprocess\n```\n\nNotes:\n- Every argument below is passed as `--env.<arg>`, e.g. `--env.taskset.db-seed 7`.\n- `--env.agent.max-turns` is unbounded by default; `--env.agent.runtime.type subprocess`\n  keeps the tool server local instead of a Prime sandbox.\n\n### Environment Arguments\n\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `taskset.buckets` | `list[str]` | `[]` | Restrict to these failure-mode buckets; empty loads all 57 tasks |\n| `taskset.db-seed` | `int` | `20260825` | Which generated database to run against |\n| `taskset.task.tools.db-path` | `str` | `\"\"` | Absolute path to a database, overrides `db-seed` |\n| `taskset.task.tools.max-rows` | `int` | `100` | Row cap on a query result |\n| `taskset.task.tools.max-chars` | `int` | `8000` | Character cap on rendered query output |\n| `taskset.task.tools.query-timeout-s` | `float` | `15.0` | Per-query timeout |\n| `taskset.task.judges` | `list[JudgeConfig]` | reference judge, `openai/gpt-5.4-nano` | Grader(s) applied to the final reply |\n| `taskset.task.rewards.partial-credit.weight` | `float` | `0.0` | Weight for the schema-coverage partial-credit reward |\n\n### Metrics\n\n| Metric | Meaning |\n| ------ | ------- |\n| `correct_answer` | Reward (weight 1.0). Judge verdict on the final reply: 1.0 correct, 0.0 incorrect |\n| `tables_covered` | Reward (weight 0.0). Fraction of the task's expected tables referenced in executed SQL |\n| `columns_covered` | Reward (weight 0.0). Fraction of the task's expected columns referenced in executed SQL |\n| `query_wellformed` | Reward (weight 0.0). 1.0 if some executed query returned without a SQL error |\n| `exact_match` | Metric. Deterministic string comparison against ground truth |\n| `exact_<bucket>` | Metric. `exact_match` split by task bucket |\n| `replied` | Metric. 1.0 if the final message was non-empty |\n| `reply_chars` | Metric. Length of the final reply |\n| `sql_queries` | Metric. Number of `sqlite_query` calls made |\n| `tool_calls` | Metric. Number of tool calls made, of any kind |\n| `query_efficiency` | Metric. `1 / sql_queries` when exact-matched, else 0 |\n","encoding":"utf-8","truncated":false,"total_bytes":3619},"status":null}