{"data":{"kind":"file","path":"README.md","version_id":"u8m64y1jasw5pov97yro7ib4","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2890,"modified_at":"2026-06-28T05:27:15.351000","content_hash":"b3fccd21a4a5a17997b3917ed1099e55e1a3738303376589c0464e87916cc12e"},"entries":[],"content":"# pico-dsl-v0\n\n### Overview\n- **Environment ID**: `pico-dsl-v0`\n- **Short description**: Single-turn, no-tool eval for executing a tiny per-sample string transduction DSL.\n- **Tags**: eval, single-turn, no-tools, dsl, string-transduction\n\n### Dataset\n- **Primary dataset**: Deterministically generated synthetic examples from the local oracle interpreter.\n- **Default size**: 240 examples.\n- **Default split**: eval-only; the same deterministic dataset is exposed as both `dataset` and `eval_dataset` for runner compatibility.\n\n### Task\n- **Type**: single-turn, no tools.\n- **Response format**: strict JSON object, `{\"answer\": \"...\"}`.\n- **Operations**: `TAKE`, `DROP`, `REPEAT`, `UPPER`, `LOWER`, `SWAPCASE`, and literal tape characters.\n- **Excluded from v0**: toggle state, reverse wrapping, and tail consumption.\n- **Rubric**: exact match on the parsed `answer`; `format_valid` is logged as a zero-weight diagnostic metric.\n\n### Quickstart\n\n```bash\nuv run vf-eval pico-dsl-v0 -m <model-or-endpoint-alias> -n 20 -r 1 -t 128 -T 0\n```\n\nWith Prime Inference:\n\n```bash\nprime --plain eval run pico-dsl-v0 -m meta-llama/llama-3.1-70b-instruct -n 20 -r 1 -t 128 -T 0\n```\n\nThis package is also registered as a local `uv` workspace member from the repository root, so `uv sync` installs it for local eval commands.\n\n### Environment Arguments\n\nPass with `-a` / `--env-args` as JSON.\n\n| Arg | Type | Default | Description |\n| --- | --- | --- | --- |\n| `num_examples` | int | `240` | Number of deterministic examples to generate. |\n| `seed` | int | `0` | RNG seed for deterministic generation. |\n| `difficulty` | str | `\"mixed\"` | One of `tiny`, `easy`, `medium`, `hard`, or `mixed`. |\n\nExample:\n\n```bash\nuv run vf-eval pico-dsl-v0 -m <model-or-endpoint-alias> -a '{\"difficulty\":\"hard\",\"num_examples\":100}' -n 100 -r 1 -t 128 -T 0\n```\n\n### Local Elsie Endpoint\n\n`configs/endpoints.toml` defines `elsie-qwen35-4b` at `http://elsie-2:8000/v1`.\nSet a dummy key before running local evals:\n\n```bash\nexport ELSIE_API_KEY=dummy\nuv run vf-eval pico-dsl-v0 -m elsie-qwen35-4b -e configs/endpoints.toml -n 20 -r 1 -t 128 -T 0 --disable-env-server\n```\n\nThe current `elsie-qwen35-4b` server follows the Qwen3.5 model-card serving note and runs vLLM with `--reasoning-parser qwen3 --language-model-only`. For no-CoT evals, pass Qwen's non-thinking request knob under OpenAI SDK `extra_body`:\n\n```bash\nuv run vf-eval pico-dsl-v0 -m elsie-qwen35-4b -e configs/endpoints.toml -n 6 -r 1 -t 80 -T 0 --disable-env-server -a '{\"difficulty\":\"tiny\",\"seed\":15,\"num_examples\":6}' -S '{\"extra_body\":{\"chat_template_kwargs\":{\"enable_thinking\":false}}}'\n```\n\n### Metrics\n\n| Metric | Meaning |\n| --- | --- |\n| `reward` | Same as exact match. |\n| `exact_match` | `1.0` when parsed JSON `answer` exactly matches the oracle output. |\n| `format_valid` | `1.0` when the model returned a valid JSON object with a string `answer`. |\n","encoding":"utf-8","truncated":false,"total_bytes":2890},"status":null}