{"data":{"kind":"file","path":"README.md","version_id":"agp90rbl6jy5m5m8g8lj0tmu","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5941,"modified_at":"2025-08-22T15:12:31.466000","content_hash":"f5664a663ce1c61c973e91c36dd4ee60c4184e243925a2a0603c05157f6503c5"},"entries":[],"content":"# syllabify-en\r\n\r\n### Overview\r\n\r\n- **Environment ID**: `syllabify-en`\r\n- **Short description**: Proto environment for syllable breaking of English. Uses sonority sequencing.\r\n- **Tags**: syllabification, tokenization, NLP, sonority, label-free\r\n\r\n### Motivation\r\n\r\nThis may seem like a toy task, but it exploits a weakness in current tokenizer-based models: to the model, the token orthography (spelling)\r\nis opaque, and there's generally no reason to expect token boundaries to align with syllable boundaries.\r\n\r\nSo for example, if it gets \"bullseye\" as input corresponding to token 123 and needs to generate \"bulls.eye\" corresponding to tokens [456, 78, 910],\r\nit needs to somehow know:\r\n\r\n - 123 has eight letters\r\n - 123 has two syllables\r\n - the first syllable of 123 is five letters long\r\n - the second syllable of 123 is three letters long\r\n - 456 has five letters\r\n - 456 corresponds to the first five letters of 123\r\n - 910 has three letters\r\n - 910 corresponds to the last three letters of 123\r\n\r\nBut none of this information is explicitly available to the model.\r\n\r\n### Datasets\r\n\r\n- **Primary dataset(s)**: Default hardcoded samples or any HuggingFace dataset via configuration\r\n- **Source links**: Configurable via `hf_dataset_id` parameter\r\n- **Split sizes**: Depends on selected dataset (uses first available split)\r\n\r\n### Task\r\n\r\n- **Type**: single-turn\r\n- **Parser**: custom\r\n- **Rubric overview**: Single Criterion: Symmetric Damerau-Levenshtein Similarity\r\n\r\n### Quickstart\r\n\r\nRun an evaluation with default settings:\r\n\r\n```bash\r\nuv run vf-eval syllabify-en\r\n```\r\n\r\nConfigure model and sampling:\r\n\r\n```bash\r\nuv run vf-eval syllabify-en -m gpt-4.1-mini -n 4 -r 3 -t 1024 -T 0.7\r\n```\r\n\r\nUse with a HuggingFace dataset:\r\n\r\n```bash\r\nuv run vf-eval syllabify-en -a '{\"hf_dataset_id\": \"google/frames-benchmark\", \"column\": \"Prompt\", \"sep\": \".\"}'\r\n```\r\n\r\nNotes:\r\n\r\n- Use `-a` / `--env-args` to pass environment-specific configuration as a JSON object.\r\n- Reports are written under `./environments/syllabify_en/reports/` and auto-embedded below.\r\n- When using HuggingFace datasets, the environment will automatically syllabify text from the specified column using NLTK's sonority sequencing algorithm.\r\n\r\n### Environment Arguments\r\n\r\n- `hf_dataset_id` (optional): HuggingFace dataset identifier to load instead of using default samples\r\n- `column` (optional): Column name in the dataset containing text to syllabify (required if hf_dataset_id is provided)\r\n- `sep` (optional): Separator to use between syllables (default: \".\")\r\n\r\n### Metrics\r\n\r\n| Metric | Meaning |\r\n| ------ | ------- |\r\n| `indel_accuracy` | Main scalar reward, Damerau-Levenshtein Similarity |\r\n\r\n## Evaluation Reports\r\n\r\n\r\n\r\n
Reports\r\n
syllabify-en--v0.1.0--model=gpt-4.1-mini--n=4--r=3--args=noargs\r\n

Open full report

\r\n

syllabify-en: gpt-4.1-mini (n=4, r=3)

\r\n
\r\n
Environment: syllabify-en (v0.1.0)
\r\n
Model: gpt-4.1-mini
\r\n
Provider: https://api.openai.com/v1
\r\n
Samples: n=4, r=3
\r\n
Date: 2025-08-22
\r\n
Time: 11:12:31
\r\n
Sampling: max_tokens=1024, temperature=0.7
\r\n
\r\n\r\n

Reward

\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
meanstdnp5p25p50p75p95
0.97370.0461120.88680.97811.01.01.0
\r\n\r\n\r\n

Metrics

\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
metricmeanstdnp5p25p50p75p95
indel_accuracy0.97370.0461120.88680.97811.01.01.0
\r\n\r\n\r\n

Examples (showing up to 12 of 12)

\r\n
\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
#rewardindel_accuracycompletion
01.01.0
co-quet-tish, flam-boy-ant and chur-lish
11.01.0
the quick brown fox jumps o.ver the la.zy dog
21.01.0
the Rain in spain Stays main<SEP>ly in the plain
30.89470.8947
gre🪲gor sams🪲a a🪲woke one morn🪲ing from un🪲e🪲a🪲sy dreams
41.01.0
co-quet-tish, flam-boy-ant and chur-lish
51.01.0
the quick brown fox jumps o.ver the la.zy dog
61.01.0
the Rain in spain Stays main<SEP>ly in the plain
70.91230.9123
gre🪲gor samsa a🪲woke one morn🪲ing from un🪲ea🪲sy dreams
81.01.0
co-quet-tish, flam-boy-ant and chur-lish
91.01.0
the quick brown fox jumps o.ver the la.zy dog
101.01.0
the Rain in spain Stays main<SEP>ly in the plain
110.87720.8772
gre🪲gor samsa a🪲woke one morn🪲ing from un🪲ease🪲y dreams
\r\n
\r\n
\r\n
\r\n\r\n","encoding":"utf-8","truncated":false,"total_bytes":5941},"status":null}