{"data":{"kind":"file","path":"README.md","version_id":"rlmwvk0kwifiyddnpz9t74w8","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5066,"modified_at":"2026-07-01T18:45:54.916000","content_hash":"45785e42b5c92941a598d53b235aa5ee1c992f9aa4ab09eec0d46e05f6dabd3b"},"entries":[],"content":"# pico-dsl-v1\n\n### Overview\n- **Environment ID**: `pico-dsl-v1`\n- **Short description**: Four-line reverse-then-transduce string eval for a tiny per-sample DSL.\n- **Tags**: eval, single-turn, no-tools, dsl, string-transduction, reverse, synthetic-phrases\n\n### Dataset\n- **Primary dataset**: A committed 480-row fixed synthetic benchmark at `data/pico_dsl_v1_benchmark.jsonl`.\n- **Generation path**: Offline NVIDIA NeMo Data Designer phrase seeds, deterministic tape compilation, and strict local validation.\n- **Default size**: 480 examples.\n- **Default split**: eval-only; the same deterministic dataset is exposed as both `dataset` and `eval_dataset` for runner compatibility.\n- **Ad hoc mode**: `dataset_mode=\"generated\"` keeps the old random generator available for debugging, with legacy `difficulty` labels.\n\n### Fixed Slices\n\nThe primary public selector is `slice`, not `difficulty`.\n\n| Slice | What it targets |\n| --- | --- |\n| `copy_case` | Source-driven copying plus case transforms. |\n| `punctuation` | Literal punctuation/bracket/spacing insertion around source-driven text. |\n| `deletion` | `DROP` behavior with all source characters consumed. |\n| `repeat` | A contributing `REPEAT` operation. |\n| `noop_suffix` | Extra consuming instructions after source exhaustion that must not change output. |\n| `composite` | At least three primitive phenomena in the same row. |\n\nEach slice has 80 rows, balanced across diagnostic `complexity_level` values `L1`-`L4` with 20 rows per cell. `slice=\"mixed\"` selects rows round-robin across the 6 x 4 grid.\n\n### Task\nThe model receives an input string and an instruction tape. It must put exactly four newline-separated lines inside one `<answer>...</answer>` block:\n\n1. the original input string\n2. the character-by-character reverse of the input string\n3. the result of applying the tape to the reversed input string\n4. the character-by-character reverse of line 3\n\nNo text is allowed outside the `<answer>` block. No JSON, labels, Markdown, code fences, or prose are allowed inside it. The prompt tells models to use the `<answer>` tag only when ready to submit the final answer, and the scorer only treats a trailing `<answer>...</answer>` block as final, so quoted or drafted tags inside reasoning are ignored.\n\n### Operations\nThe per-sample dictionary maps random symbols to `TAKE`, `DROP`, `REPEAT`, `UPPER`, `LOWER`, and `SWAPCASE`. Tape characters not present in the dictionary are literals.\n\n### Rubric\nThe headline reward is strict all-line exact match. Diagnostic metrics include:\n\n| Metric | Meaning |\n| --- | --- |\n| `exact_match` | `1.0` when all four parsed lines exactly match the oracle. |\n| `format_valid` | `1.0` when the response ends with an `<answer>` block that parses as exactly four lines. |\n| `answer_tag_valid` | `1.0` when a trailing `<answer>` block is present. |\n| `line_count_ok` | `1.0` when the parsed answer, or fallback visible response, has exactly four lines after removing final newlines. |\n| `line_1_exact` ... `line_4_exact` | Per-line exact-match diagnostics. |\n| `avg_line_similarity` | Mean per-line normalized Levenshtein similarity. |\n| `line_1_similarity` ... `line_4_similarity` | Per-line normalized edit similarity. |\n\nThe normalized edit-distance metrics are zero-weight diagnostics, not the reward.\n\n### Quickstart\n\n```bash\nuv run vf-eval pico-dsl-v1 -m <model-or-endpoint-alias> -n 20 -r 1 -t 256 -T 0\n```\n\nWith fixed benchmark args:\n\n```bash\nuv run vf-eval pico-dsl-v1 -m <model-or-endpoint-alias> -a '{\"slice\":\"repeat\",\"num_examples\":80}' -n 80 -r 1 -t 256 -T 0\n```\n\nWith ad hoc generated args:\n\n```bash\nuv run vf-eval pico-dsl-v1 -m <model-or-endpoint-alias> -a '{\"dataset_mode\":\"generated\",\"difficulty\":\"hard\",\"num_examples\":100}' -n 100 -r 1 -t 256 -T 0\n```\n\n### Environment Arguments\n\nPass with `-a` / `--env-args` as JSON.\n\n| Arg | Type | Default | Description |\n| --- | --- | --- | --- |\n| `num_examples` | int | `480` | Number of fixed benchmark examples to select. |\n| `slice` | str | `\"mixed\"` | One of `mixed`, `copy_case`, `punctuation`, `deletion`, `repeat`, `noop_suffix`, or `composite`. |\n| `dataset_mode` | str | `\"fixed\"` | `fixed` for the committed benchmark or `generated` for the old ad hoc generator. |\n| `seed` | int | `0` | Used only by `dataset_mode=\"generated\"`. |\n| `difficulty` | str | `null` | Legacy generated-mode selector: `tiny`, `easy`, `medium`, `hard`, `mixed`, or the old tape-length aliases. |\n| `tokenizer` | str | `\"auto\"` | Optional generated-mode tiktoken selector. |\n\n### Regenerating Fixed Data\n\n```bash\nuv run python scripts/generate_pico_dsl_v1_benchmark.py\n```\n\nThe builder uses Data Designer against `http://elsie-1:8000/v1` and model `elsie-qwen35-27b-atlas` by default, then writes only rows that pass the shared oracle validator.\n\nThe validator also enforces content and tape-quality gates: balanced 6 x 4 slice/level coverage, no duplicate short normalized targets, no filler-like target tails, no lowercase `ml` token, and per-slice entropy/unique-character floors so low-entropy tapes remain rare exceptions.\n","encoding":"utf-8","truncated":false,"total_bytes":5066},"status":null}