{"data":{"kind":"file","path":"README.md","version_id":"wufwnrma8lov20ckqhhyyk9o","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5614,"modified_at":"2026-11-05T00:31:10","content_hash":"1f94a69e3a1b25ca95c2944061310c1eed15607be210107e3b38ab3a8992d2ce"},"entries":[],"content":"# bird-sql\n\nA verifiers v1 environment for single-turn text-to-SQL on [BIRD](https://bird-bench.github.io/). Each task is one BIRD question. The model gets the database schema, BIRD's hint (\"evidence\") and the question, and answers with one SQLite query in a ```` ```sql ```` block. The environment runs that query and the gold query on the real BIRD database and rewards the model when the returned rows match.\n\nIt is the evaluator and reward from [text2sql-grpo](https://github.com/HassanDawy/text2sql-grpo), where it was used to train Qwen2.5-Coder-1.5B-Instruct with SFT and then GRPO.\n\n## Reward and metrics\n\n| Name | Kind | Meaning |\n| --- | --- | --- |\n| `execution_match` | reward (weight 1) | 1 if the answer's rows equal the gold rows as sets (BIRD's official rule, so row order and `1` vs `1.0` don't matter), else 0 |\n| `error` | metric | 1 if there was no ```` ```sql ```` block, the SQL failed, or it timed out |\n| `trivial` | metric | 1 if the answer returned no rows or a single 0, 1 or NULL value |\n| `gold_error` | metric | 1 if the gold query itself failed (the answer then scores 0) |\n\nMetrics are recorded but not added to the reward.\n\n**Why no partial credit.** An earlier design gave 0.1 for SQL that runs but returns wrong rows. That makes any runnable query at least as good as an honest attempt that errors, which invites reward hacking. The reward is binary, as in the SLM-SQL paper.\n\n**Why `trivial` exists.** About 7% of BIRD train questions have a gold answer of a single 0/1, a single NULL, or no rows. On those, a query like `SELECT 1` scores 1. Track `trivial` during training: if it rises while accuracy stays flat, the policy is learning shortcuts.\n\n## Setup\n\n1. Download BIRD dev (and train if you need it) from the [BIRD site](https://bird-bench.github.io/) and unzip the databases, so the folder looks like:\n\n   ```\n   dev_20240627/\n     dev.json\n     dev_databases/dev_databases/<db_id>/<db_id>.sqlite\n   ```\n\n   BIRD data is not included in this package. Check BIRD's license before redistributing it.\n\n2. Install the environment (Python 3.11+, Linux or macOS; verifiers does not run on native Windows):\n\n   ```bash\n   pip install -e environments/bird_sql\n   ```\n\n3. Point it at the data with `--env.taskset.data-dir <folder>`, or set `$BIRD_DIR` (dev) / `$BIRD_TRAIN_DIR` (train).\n\n## Run an evaluation\n\nThis is a single-turn task, so use the plain `null` harness (no tools) and run it locally:\n\n```bash\nvf-run bird-sql -m <model> \\\n  --client.base-url <OpenAI-compatible URL> \\\n  --client.api-key-var <ENV_VAR_WITH_KEY> \\\n  --sampling.temperature 0 \\\n  --env.agent.harness.id null \\\n  --env.agent.runtime.type subprocess\n```\n\nWithout the last two flags, verifiers 0.4.1 defaults to the `bash` coding-agent harness in a Prime cloud sandbox. That adds a shell tool to the prompt and needs a Prime login.\n\nOptions:\n\n- `--env.taskset.split dev|train` (default `dev`). Train skips the 377 questions whose gold SQL fails on SQLite 3.35.5 (listed in `bird_sql/train_gold_failures.txt`; ids are positions in `train.json`).\n- `--env.taskset.num-tasks N` uses the first N questions.\n- `BIRD_SQL_WORKERS` sets how many SQL queries run at once (default: one per CPU core the container may use). A query waits for a free slot before its 30 s timeout starts, so a busy machine queues instead of timing out.\n\n**Serving your own model with vLLM:** start the server with `--generation-config vllm`. By default, `vllm serve` applies the sampling defaults from the model's `generation_config.json`. Qwen2.5 models carry `repetition_penalty: 1.1` there, which silently changes greedy outputs. In our runs it cost about 2 points of accuracy on BIRD dev.\n\n## Scoring details\n\n- Databases are opened read-only, so model SQL cannot modify them.\n- Timeouts: 30 s for the model's query, 120 s for the gold query.\n- Answer rows are streamed and compared against the gold set, so huge results don't run out of memory.\n- Only the first ```` ```sql ```` block is used.\n- SQLite version matters for speed. Reference numbers were scored on SQLite 3.35.5. On SQLite 3.49.1 some gold queries were far slower (one went from 0.2 s to over 600 s) and would time out.\n\n## Baselines (BIRD dev, 1,534 questions)\n\nPrompt as in this environment, one greedy answer per question (temperature 0), scored on SQLite 3.35.5.\n\n| Model | Execution accuracy |\n| --- | --- |\n| GPT-4o-mini (2024-07-18) | 50.5% |\n| Qwen2.5-Coder-1.5B-Instruct, zero-shot | 26.5% |\n| + SFT (LoRA r16) | 44.4% |\n| + GRPO with this reward (LoRA r16) | 47.5% |\n\nRunning GRPO and GPT-4o-mini through this environment with `vf-run` gave 47.8% and 50.8%. The answer text was identical on 95% (GRPO via vLLM) and 78% (GPT-4o-mini) of questions, and every identical answer got the same score. The remaining gap is decoding noise.\n\nThese numbers use a short, direct prompt and a single sample. Published results on the same model use reasoning-first prompts and self-consistency over many samples, so they are not directly comparable.\n\n## Related environments\n\nOther text-to-SQL environments on the Environments Hub:\n\n- [sql-single-turn](https://app.primeintellect.ai/dashboard/environments/stelioszach/sql-single-turn): single-turn, synthetic templated schemas.\n- [skyrl-sql](https://app.primeintellect.ai/dashboard/environments/primeintellect/skyrl-sql): multi-turn, following the SkyRL-SQL protocol, evaluated on Spider.\n\nBoth use the legacy verifiers v0 API. This environment uses v1 and BIRD.\n\n## Citation\n\nBIRD: Li et al., *Can LLM Already Serve as a Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs*, NeurIPS 2023.\n","encoding":"utf-8","truncated":false,"total_bytes":5614},"status":null}