{"data":{"kind":"file","path":"README.md","version_id":"zvqjr9eczqbgg9i84vmhpzvf","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":16736,"modified_at":"2026-09-14T21:55:10.764000","content_hash":"49a98e36a24aafad2c69bca5c59e2c976391e9f1f56d605de90a6aac5b0691e3"},"entries":[],"content":"# instruction-adherence\n\n> **Public build.** This package contains the constraint library, rubric and environment code plus a bundled 500-instance sample of the `held_out_combos` split. Instance generation (`K`, `burial_depth`, `hard_family_weight`, other splits) is not included; `load_environment()` defaults to the bundled sample.\n\nProcedurally generated instruction-following tasks with **3-16 stacked, code-checkable\nconstraints**, in single-turn and scripted multi-turn form, with deterministic rewards and\nper-family diagnostics.\n\n- **Environment ID**: `instruction-adherence` (module `instruction_adherence`)\n- **Type**: single-turn and multi-turn (scripted user, no LLM), grouped as a `vf.EnvGroup`\n- **Constraint taxonomy**: 46 constraint types across 6 families (44 trainable, 2 `eval_only`)\n- **Seeds**: 200 seed tasks across 8 task types, each with slot fillers (176 public, 24 reserved for the private test split)\n- **Instances**: unlimited, deterministic given `(split, seed, index)`\n\nWhy it exists: public instruction-following sets have ~25 constraint types and a few hundred\nfixed prompts, and they are saturated and contaminated. This environment has a large\ntaxonomy, unlimited fresh instances, compositional held-out splits, multi-turn persistence,\nand per-family diagnostics.\n\n## Quickstart\n\n```bash\n# from the lab workspace root\nprime eval run instruction_adherence -m <model> -n 50 -r 8 \\\n  -a '{\"mode\": \"both\", \"split\": \"held_out_combos\"}' -s\nuv run python environments/instruction_adherence/scripts/report.py outputs/<run-dir>\n```\n\n`load_environment` arguments:\n\n| arg | default | meaning |\n|---|---|---|\n| `K` | calibrated (else 10) | number of constraints (3-16); in multi-turn, the maximum active at once |\n| `burial_depth` | calibrated (else 1) | 0 block list; 1 split across preamble and task body; 2 plus a trailing P.S.; 3 plus a quoted notes block |\n| `hard_family_weight` | calibrated (else 1.0) | sampling weight for lipogram, acrostic, positional, distinct-sentence-start constraints and the multi-turn long-gap pattern |\n| `mode` | `\"both\"` | `single`, `multi` or `both` (EnvGroup with sub-envs `single` / `multi`) |\n| `split` | `\"train\"` | `train`, `held_out_combos` or `test` (test needs the private seeds) |\n| `num_examples` | 500 | instances per sub-environment |\n| `seed` | 0 | instance seed |\n\nWhen `K`, `burial_depth` and `hard_family_weight` are all omitted, a pre-generated dataset in\n`data/<split>` is used if present. Otherwise instances are generated on the fly. Defaults come\nfrom `calibration.json` when it exists.\n\n## Text definitions\n\nAll checkers share `instruction_adherence/textutils.py`:\n\n- **word**: a maximal run of letters, optionally joined by internal apostrophes or hyphens (`don't`, `well-known`). Numbers and punctuation are not words.\n- **line**: a non-empty line of the response.\n- **heading**: a line starting with 1-6 `#` followed by a space. A **rule** is a line of only `---`, `***` or `___`.\n- **title**: a line of the form `<<title>>`. A **list item** is a line starting with `-`, `*`, `•`, `1.`/`1)` or `a.`/`a)`.\n- **paragraph**: a blank-line-separated block containing at least one word of prose. Heading, rule, title and list-item lines are not paragraph text, so a list is never a paragraph.\n- **sentence**: every non-heading, non-rule, non-title line (list items included) is split after `.`, `!` or `?` followed by whitespace. Each piece with at least one word is a sentence. A line break always ends a sentence, so every list item is a sentence.\n- **first letter** of a unit: its first alphabetic character (markers like `[1]`, `-`, `**` are skipped).\n- **removed before checking**: a response wrapper (`<<<`...`>>>`, `[[START]]`...`[[END]]`, `BEGIN RESPONSE`...`END RESPONSE`), a leading `REQUIREMENTS:` ... `END REQUIREMENTS` block (the closing marker must be on its own line), and a trailing `SELF-AUDIT` block. `struct.wrapped` and the meta constraints check the raw response. The prompts requesting meta blocks say so.\n\n## Constraint taxonomy\n\n| id | family | difficulty | eval_only | hard family | example phrasing |\n|---|---|---|---|---|---|\n| `length.word_range` | length | 2 | no |  | Keep the length to at least 70 and at most 120 words. |\n| `length.sentence_count` | length | 3 | no |  | Your answer must consist of precisely 13 sentences (every list item or line counts as its own sentence). |\n| `length.paragraph_count` | length | 2 | no |  | Organize the response into exactly 2 paragraphs separated by blank lines. |\n| `length.bullet_count` | length | 2 | no |  | Include exactly 6 bullet points, each on its own line starting with \"- \". |\n| `length.paragraph_word_range` | length | 3 | no |  | Every paragraph must be between 50 and 75 words long. |\n| `length.sentence_max_words` | length | 3 | no |  | Every sentence must contain fewer than 21 words. |\n| `length.nth_longest` | length | 3 | no |  | The third paragraph must be strictly the longest paragraph (by word count). |\n| `length.nth_shortest` | length | 3 | no |  | The third paragraph must be strictly the shortest paragraph (by word count). |\n| `length.increasing_paragraphs` | length | 3 | no |  | Write at least 3 paragraphs, each containing more words than the one before it. |\n| `lex.lipogram` | lexical | 5 | no | lipogram | Do not use the letter \"T\" anywhere in your response, in upper or lower case. |\n| `lex.forbidden_words` | lexical | 2 | no |  | Do not use the words \"really\", \"new\" and \"just\", including any of their inflected forms. |\n| `lex.word_exact_k` | lexical | 3 | no |  | Use the word \"together\" exactly 4 times (case-insensitive; other forms such as plurals do not count). |\n| `lex.word_order` | lexical | 2 | no |  | Use both the word \"plan\" and the word \"outcome\", and make the first \"plan\" appear before the first \"outcome\". |\n| `lex.no_repeat` | lexical | 4 | no |  | No word may appear more than twice (case-insensitive), except these common words: a, an, and, ... |\n| `lex.distinct_sentence_starts` | lexical | 4 | no | distinct_sentence_starts | Every sentence must begin with a different letter. |\n| `lex.acrostic` | lexical | 5 | no | acrostic | Write exactly 5 paragraphs whose first letters, in order, spell \"SPARK\". |\n| `lex.exact_phrase` | lexical | 1 | no |  | Include the exact phrase \"for what it is worth\" verbatim, all in lowercase. |\n| `lex.synonym_ban` | lexical | 3 | no |  | Do not use any of these words or their forms: \"show\", \"demonstrate\", \"reveal\" and \"display\". |\n| `lex.no_commas` | lexical | 2 | no |  | Do not use any commas in your response. |\n| `struct.headers` | structure | 2 | no |  | Use exactly 3 markdown headers, in this order and with this exact text: \"## Background\", \"## Summary\" and \"## Examples\". |\n| `struct.numbered_sublist` | structure | 3 | no |  | Include a numbered list of exactly 4 items; item 1, and only item 1, must contain an indented sub-list of at least two entries. |\n| `struct.wrapped` | structure | 1 | no |  | Wrap your entire response between [[START]] at the very start and [[END]] at the very end. |\n| `struct.json_keys` | structure | 3 | no |  | Respond with only a valid JSON object whose keys are exactly \"audience\", \"next_step\" and \"conclusion\" in that order. |\n| `struct.first_last_word` | structure | 3 | no |  | The first word of your response must be the same as the last word (case-insensitive). |\n| `struct.ends_question` | structure | 1 | no |  | End your response with a question: the final character must be \"?\". |\n| `struct.alternating_lines` | structure | 4 | no |  | At least 6 lines alternating long (12+ words) and short (1-5 words). |\n| `struct.sections` | structure | 2 | no |  | Divide the response into exactly 3 sections separated by a line containing only ---. |\n| `struct.title` | structure | 1 | no |  | Start with a title on its own first line, wrapped in double angular brackets, like <<my title>>. |\n| `struct.bold_count` | structure | 2 | no |  | Bold exactly 4 phrases using **double asterisks**; no other bold text. |\n| `lang.paragraph_language` | language | 3 | no |  | Write the first paragraph entirely in French; every other paragraph must be in English. |\n| `lang.lowercase` | language | 3 | no |  | Write in lowercase letters only; the only words allowed to contain capital letters are these proper nouns: Priya. |\n| `lang.no_contractions` | language | 2 | no |  | Do not use contractions, and avoid possessive 's too; no word may contain an apostrophe. |\n| `lang.second_person` | language | 3 | no |  | Address the reader directly as \"you\" (at least 3 times) and never use first-person pronouns. |\n| `lang.present_tense` | language | 4 | **yes** |  | Write entirely in the present tense. |\n| `lang.no_passive` | language | 4 | **yes** |  | Do not use the passive voice anywhere. |\n| `lang.caps_limit` | language | 1 | no |  | No more than one ALL-CAPS word (two letters or longer) may appear. |\n| `pos.word_in_paragraph` | positional | 3 | no | positional | The word \"remarkably\" must appear in the first paragraph and in no other paragraph. |\n| `pos.date_after_event` | positional | 3 | no | positional | Every time you use the word \"launch\" (or \"launches\"), an ISO-8601 date must follow it later in the same sentence. |\n| `pos.topic_order` | positional | 3 | no | positional | The first paragraph mentioning \"flour\" must come before the first paragraph mentioning \"bakery\". |\n| `pos.mention_ratio` | positional | 3 | no | positional | Sentences mentioning \"bakery\" must be at least twice the sentences mentioning \"oven\". |\n| `pos.however_once` | positional | 2 | no | positional | Exactly one sentence must begin with the word \"however\". |\n| `pos.final_paragraph_start` | positional | 2 | no | positional | Begin the final paragraph with the exact words \"In short\". |\n| `pos.nth_sentence_contains` | positional | 2 | no | positional | The fourth sentence of your response must contain the word \"therefore\". |\n| `meta.self_audit` | meta | 3 | no |  | End with a SELF-AUDIT block listing R<number>: yes/no for every labelled requirement. |\n| `meta.restate` | meta | 3 | no |  | Restate every other requirement word for word between REQUIREMENTS: and END REQUIREMENTS before answering. |\n| `meta.number_paragraphs` | meta | 2 | no |  | Number each paragraph in square brackets at its start: [1], [2], [3], and so on. |\n\nEvery constraint has 3-4 phrasing templates, and parameters are sampled per instance.\nCheckers never accept a response that violates the rendered instruction. The two heuristic\ncheckers (`present_tense`, `no_passive`) are `eval_only` and only appear in `test`.\n\n## Sampling and contradiction avoidance\n\n`compat.py` rejects contradictory sets in three layers:\n\n1. an explicit tag conflict graph (e.g. `json_output` vs. markdown headers or paragraphs, `short` vs. `multi_paragraph`, and two constraints that both pin the start or end of the response);\n2. parameter rules: literal strings one constraint requires but another forbids (e.g. a required header containing a banned letter), plus per-class conflicts;\n3. a satisfiability smoke test: merge numeric bounds and constructively synthesize a text that passes every count-based constraint, resampling if none is found.\n\nAt K=16, 10,000 sampled sets contain zero contradictions, with a 21.6% draw rejection rate (`tests/test_compat.py`).\n\n## Splits\n\nSplits are compositional at the level of constraint-id pairs:\n\n| split | size | seeds | constraints | guarantee |\n|---|---|---|---|---|\n| `train` | 10,000 | public | trainable only | never contains any of 40 held-out or 40 test pairs |\n| `held_out_combos` | 1,000 | public | trainable only | every instance contains a held-out pair, so its combination never appears in train |\n| `test` | 1,000 | private (24 seeds) | includes `eval_only` | disjoint seeds and a test pair in every instance; **never leaves the machine** |\n\nRegenerate with `uv run python environments/instruction_adherence/scripts/gen_dataset.py`.\nRows are ~30% multi-turn, with K ~ U{3..16}, depth ~ U{0..3} and hard_family_weight ~ U{1,2,3}.\n`data/train` and `data/test` are gitignored and excluded from Hub builds. The public package\nbundles a 500-instance sample of `held_out_combos`.\n\n## Multi-turn scripts\n\nScripts are 4-12 turns long, and each user turn brings a new writing task. Each script mixes\nseveral patterns:\n\n- **persistence**: turn-1 constraints stay active with no reminder;\n- **addition**: new constraints are added mid-conversation;\n- **revocation**: an observable constraint is dropped, after which the response must *not* satisfy it;\n- **contradiction**: a conflicting constraint is added. That turn is scored without either constraint and rewards flagging the conflict. The next turn restores the original;\n- **distractor**: a quoted instruction meant for someone else, which must *not* be adopted;\n- **long gap**: turns 8+ test constraints set 7 or more turns earlier.\n\nEvery assistant turn is scored against the constraint set active at that turn.\n\n## Rewards\n\n| function | weight | notes |\n|---|---|---|\n| `all_satisfied` | 1.0 | binary per turn; mean over turns in multi-turn |\n| `fraction_satisfied` | 0.3 | fraction of active checks passed |\n| `task_relevance` | 0.2 | coverage of usable topic keywords (keywords banned by a constraint are ignored) |\n| `no_leakage` | 0.1 | 1 below 20% word-4-gram overlap with the requirement texts, decaying to 0 at 50% |\n| `self_audit_accuracy` | 0.2 | F1 of claimed vs. actual satisfaction (only with `meta.self_audit`) |\n| `conflict_flagged` | 0.5 | multi-turn contradiction turns only |\n| `content_floor` | gate | under 30 words scores 0 everywhere, unless a length constraint caps the response below 30 words |\n\nAdditional gates: 50% or more copied requirement text zeroes the turn; `all_satisfied` and\n`fraction_satisfied` also require at least one topic keyword.\n\nDiagnostics are emitted on every rollout as count/pass pairs: `n_<id>` / `pass_<id>` per\nconstraint, `n_fam_<family>` / `pass_fam_<family>`, `n_pat_<pattern>` / `pass_pat_<pattern>`,\n`n_turn_<i>` / `pass_turn_<i>`, plus `episode_all_satisfied` (all turns passed; used for pass@k).\n`scripts/report.py` aggregates these into pass@1/pass@8, per-family and per-constraint pass\nrates (worst first), and multi-turn pass rates by pattern and turn index.\n\n## Calibration\n\n**Single-turn: calibrated.** Default cell: `K=13, burial_depth=1, hard_family_weight=1.0`\n(`calibration.json`, generated 2026-09-14). The target was frontier pass@1 ≈ 25% / pass@8 ≈ 60%, with\nthe mid-tier model at least 10 points lower. Each cell ran 25 held-out instances × 8 rollouts with a\n16k output cap.\n\n| model (exact id) | settings | K | depth | pass@1 | pass@8 | mean reward | truncation | errors |\n|---|---|---|---|---|---|---|---|---|\n| `anthropic/claude-sonnet-5` | reasoning_effort=medium | 10 | 1 | 43.0% | 72.0% | 0.983 | 7.5% | 0% |\n| `anthropic/claude-sonnet-5` | reasoning_effort=medium | **13** | **1** | **29.0%** | **60.0%** | 0.816 | 12.5% | 0% |\n| `qwen/qwen3.6-35b-a3b` | default reasoning | **13** | **1** | **6.0%** | **16.0%** | 0.379 | 21.5% | 18.5% |\n\nqwen's errored rollouts (reasoning hitting the 16k cap) count as failures; on its non-errored rollouts\nalone pass@1 is ~7%, so the 23-point gap holds.\n\n**Multi-turn: not calibrated.** One cell was run and spending was then stopped to control cost\n(~$0.50 per multi-turn rollout). At K=10, depth 1, `claude-sonnet-5` passed 10.8% of individual turns\nand 0% of whole ~8-turn conversations, well below the band; 14% of rollouts errored on turn 1 from\nreasoning truncation. A cheaper multi-turn plan is in `NOTES.md`. Multi-turn cells are judged on mean\nper-turn pass rate (whole-conversation pass@1/pass@8 are also reported).\n\nReproduce with `uv run python environments/instruction_adherence/scripts/calibrate.py` (cached cells are\nreused); full grid in `reports/calibration.md`.\n\n## Per-family diagnostics\n\nSingle-turn default cell (K=13, depth 1), pass rate per family, worst first:\n\n| family | `claude-sonnet-5` | `qwen3.6-35b-a3b` |\n|---|---|---|\n| length | 71.1% | 35.5% |\n| meta | 69.7% | 50.7% |\n| lexical | 77.8% | 51.8% |\n| positional | 80.0% | 41.2% |\n| language | 84.4% | 53.5% |\n| structure | 86.5% | 50.4% |\n\nLength constraints (exact sentence counts, paragraph word ranges, paragraph ordering) are the weakest\nfamily for both models; positional constraints separate the two models most (38.8 points). Per-constraint\nrates are in `reports/calibration.md`, and the 50-instance frontier spot-check is in\n`reports/spotcheck_sonnet5/report.md`.\n\n## Tests\n\n```bash\ncd environments/instruction_adherence && uv run pytest -q\n```\n\nThe live smoke test (`tests/test_env_smoke.py::test_live_eval`) runs only when `PRIME_API_KEY`\nis set and `configs/endpoints.toml` lists a model.\n","encoding":"utf-8","truncated":false,"total_bytes":16736},"status":null}