{"data":{"kind":"file","path":"README.md","version_id":"j0c1d2pc6qqy7li0hd1givzj","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":7573,"modified_at":"2026-11-10T14:20:28.096000","content_hash":"a3fc6f4f78b514ecddbd41bbb457c7ea089adc8e53c24062212be37723f2f605"},"entries":[],"content":"# riggs1890-lexgym\r\n\r\nOne dictionary volume, compiled into a reinforcement-learning environment.\r\n\r\n- **Source:** Stephen R. Riggs, *A Dakota-English Dictionary* (1890), public domain.\r\n- **Tasks:** 68,911 training tasks and 5,884 evaluation tasks, in 8 task types.\r\n- **Rewards:** computed by code from the final answer line. No LLM judge.\r\n- **Provenance:** every task carries the entry id, the scan page number, and the page image name.\r\n- **Code and data pipeline:** https://github.com/HarleyCoops/Dakota1890 (`lexgym/`)\r\n\r\n## Design\r\n\r\n### 1. From scans to a lexicon graph\r\n\r\nA vision model read 336 scan pages into 24,224 structured entries. Riggs printed the dictionary in two halves, and the graph keeps both:\r\n\r\n| Half | Scan pages | Entries |\r\n|---|---|---|\r\n| Dakota → English | 95–370 | 16,980 |\r\n| English → Dakota | 371–430 | 7,244 |\r\n\r\nThe graph builder applies four rules.\r\n\r\n1. **Canonical orthography.** The vision model wrote one printed letter with several Unicode characters. A fixed table maps the variants to one form. A form that still has a character outside the Riggs alphabet stays in the graph, but it is never a gold answer.\r\n2. **Trusted headwords only.** A headword is used for the syllable task only if its printed syllable division and its single accent mark parse without error (16,227 of 16,980).\r\n3. **Two witnesses.** A Dakota form in the English half that is also a headword in the Dakota half has two independent readings. English→Dakota tasks accept only these forms.\r\n4. **Word families.** A headword and all words derived from it are one family (12,135 families). One family in ten is held out. No held-out family appears in a training row.\r\n\r\n### 2. Task types\r\n\r\n| Task | Turns | The model gets | The model must give | Reward |\r\n|---|---|---|---|---|\r\n| `syllabify` | 1 | a plain word | Riggs' syllable division and accent (`a-hi'-yo-ka-sin`) | 1.0 if exact. Otherwise 0.6 × boundary F1 + 0.2 if the stress position is right. 0 if any letter changes. |\r\n| `pos` | 1 | a word | Riggs' part-of-speech abbreviation | 1.0 if it matches, else 0 |\r\n| `inflect` | 1 | a word and its part of speech | the inflected forms that Riggs lists | F1 over the set of forms |\r\n| `root` | 1 | a derived word | the word or words it is derived from | F1 over the set of forms |\r\n| `gloss` | 1 | a Dakota word | the English definition | F1 over content words |\r\n| `en_to_dak` | 1 | an English word and its part of speech | a Dakota word | 1.0 if in the accepted set. 0.7 if only n / ŋ differs. Otherwise 0.25 × character-bigram overlap. |\r\n| `lookup_gloss` | up to 4 | a Dakota word and a dictionary tool | the English definition | F1 over content words, on the final answer only |\r\n| `lookup_root_gloss` | up to 4 | a derived word and a dictionary tool | the English definition of the word it is derived from | F1 over content words, on the final answer only |\r\n\r\nCounts:\r\n\r\n| Task | Train | Held-out families | Recall |\r\n|---|---|---|---|\r\n| `syllabify` | 13,990 | 1,446 | |\r\n| `pos` | 13,985 | 1,443 | |\r\n| `gloss` | 13,562 | | 300 |\r\n| `lookup_gloss` | 13,562 | 1,401 | |\r\n| `inflect` | 4,936 | 482 | |\r\n| `en_to_dak` | 3,369 | | 300 |\r\n| `root` | 3,060 | 287 | |\r\n| `lookup_root_gloss` | 2,447 | 225 | |\r\n\r\n### 3. The lookup tool\r\n\r\nIn the two lookup task types the model may reply with one line, `LOOKUP: <word>`. The environment returns the printed entry as the next user message:\r\n\r\n```\r\nENTRY: ya-źuŋ', v. a., to pull up by the roots with the mouth, as birds do corn [forms: mdaźuŋ] (scan page 355)\r\n```\r\n\r\nThe model has 3 lookups. `lookup_root_gloss` needs two: the first entry names the root, and the second entry holds the answer. An episode that ends on a `LOOKUP` line has no answer and scores 0. A base model can learn these tasks with no knowledge of Dakota, and they also work for word families that were held out of training.\r\n\r\n### 4. Reward rules that hold for every task\r\n\r\n- Only the final answer span is scored: the text after the last `ANSWER:`, or the last non-empty line if there is none. Gold text inside reasoning earns nothing. An unclosed `<think>` block has no answer.\r\n- The reward is in [0, 1]. Nothing multiplies it except the length guard: an answer span more than 3 times the length of the gold answer (and over 24 characters) is scaled by 3 ÷ ratio. This stops word stuffing.\r\n- There is no difficulty multiplier and no hint channel.\r\n\r\n### 5. Metrics\r\n\r\n| Metric | Weight | Meaning |\r\n|---|---|---|\r\n| `lexgym_reward` | 1.0 | the training reward |\r\n| `exact_match` | 0 | 1.0 if the answer is exactly correct |\r\n| `parse_ok` | 0 | 1.0 if the model wrote an `ANSWER:` line |\r\n| `length_mult` | 0 | the length guard multiplier |\r\n| `lookups_used` | 0 | number of dictionary reads |\r\n| `target_entry_seen` | 0 | 1.0 if the model read the entry that holds the answer |\r\n\r\n### 6. Two evaluation splits with different meanings\r\n\r\n- `heldout_rule` uses word families that training does not contain, and only task types that a rule or the lookup tool can answer. It measures generalization.\r\n- `recall` uses training vocabulary. A new word cannot be guessed, so this split measures memory of the volume.\r\n\r\n## Use\r\n\r\n```bash\r\n# from a checkout of the repository\r\nuv pip install -e environments/riggs1890_lexgym\r\n# from the Environments Hub\r\nprime env install harleycooper/riggs1890-lexgym\r\n```\r\n\r\n```python\r\nimport verifiers as vf\r\n\r\nenv = vf.load_environment(\"riggs1890-lexgym\")                       # all tasks\r\nenv = vf.load_environment(\"riggs1890-lexgym\", tasks=[\"lookup_gloss\", \"lookup_root_gloss\"])\r\n```\r\n\r\n| Argument | Default | Meaning |\r\n|---|---|---|\r\n| `tasks` | all eight | task types to keep |\r\n| `max_train_per_task` | -1 | cap on training tasks per type |\r\n| `max_eval_per_task` | 50 | cap on evaluation tasks per type and split |\r\n| `eval_splits` | `(\"heldout_rule\", \"recall\")` | evaluation splits to load |\r\n| `seed` | 42 | shuffle seed |\r\n\r\n## First results\r\n\r\nOne pilot on `Qwen/Qwen3.5-4B` (LoRA rank 32, Thinking Machines Tinker), 40 evaluation tasks per type, one seed. Exact-match rate:\r\n\r\n| Task | Split | Base model | After supervised stage | After supervised + 30 RL steps |\r\n|---|---|---|---|---|\r\n| `syllabify` | held-out | 0.00 | 0.83 | 0.83 |\r\n| `pos` | held-out | 0.05 | 0.38 | 0.73 |\r\n| `root` | held-out | 0.00 | 0.48 | 0.58 |\r\n| `inflect` | held-out | 0.00 | 0.30 | 0.38 |\r\n| `lookup_gloss` | held-out | 0.50 | 0.00 | 0.85 |\r\n| `lookup_root_gloss` | held-out | 0.10 | 0.00 | 0.63 |\r\n\r\nRL from a base model does not learn the single-turn lexical tasks, because the model cannot sample a word that it does not know. Use a supervised warm start on the same volume first (`lexgym/train_sft.py` in the repository), then RL.\r\n\r\n## Tests\r\n\r\n`tests/e2e_scripted_model.py` runs the environment end to end against a scripted OpenAI-compatible server with four policies: an oracle (reward 1.0 on all eight task types, with 1 and 2 lookups on the lookup tasks), a wrong answer, the gold answer hidden in a thinking block (reward 0), and a model that only sends `LOOKUP` (reward 0 after 3 lookups).\r\n\r\n## Limits\r\n\r\n- The source is a missionary dictionary from 1890. It records one written variety and is not a description of Dakota as it is spoken now.\r\n- The extraction has errors. 1,494 forms in the English half differ from a headword only by n / ŋ, which is likely a reading error. The scorer pays 0.7 for that difference in `en_to_dak` and records it.\r\n- A model that is correct for this book is a starting point for correction by Dakota speakers. It does not replace them.\r\n","encoding":"utf-8","truncated":false,"total_bytes":7573},"status":null}