{"data":{"kind":"file","path":"README.md","version_id":"seqhoqa6hqomntlvh129pupg","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":13407,"modified_at":"2026-09-12T11:16:49.790000","content_hash":"9f0ae8a38128836cff2e366ffd156eb8e2216da124761c77c5ca9123af8185c8"},"entries":[],"content":"# two-hop-latent-v1\n\nPort of the **semi-synthetic two-hop evaluation** from\n[\"Lessons from Studying Two-Hop Latent Reasoning\"](https://arxiv.org/abs/2411.16353)\n(Balesni, Korbak, Evans), upstream\n[`mbalesni/synthetic-two-hop`](https://github.com/mbalesni/synthetic-two-hop) pinned to\ncommit `7e15bd1`.\n\nThe paper asks whether models can compose two facts — *\"X's favorite ancient city is\nRome\"* + *\"Rome is in Europe\"* — **latently**, without writing the bridge entity down.\n18 domains of invented-person → real-entity facts, three conditions per relation: `cot`\n(reason step by step, then answer), `nocot` (answer immediately) and `nocot_shuffled`, the\npaper's control (see *Conditions*). 1,380 tasks per condition (6,900 in `nocot_shuffled`);\n69 relation files. 18 domains are shipped; the paper's main-experiment script runs 17 of\nthem (`observatories` is present in the data but omitted from that run list).\n\n## What this measures\n\nUpstream fine-tunes `meta-llama/Meta-Llama-3-8B-Instruct` on each domain's\n`train/first_hop.jsonl` — the invented first-hop facts, \"X's favorite ancient city is\nRome\" — and only then evaluates the test files. This port ships the evaluation half only.\nThe first-hop facts exist nowhere but in that training file, so for a model that has not\nlearned them `correct` is a prior/guessing baseline (several relations, `continent` for\none, are small closed sets) and must not be read as latent two-hop reasoning. Intended\nuses:\n\n1. evaluate a model fine-tuned on `train/first_hop.jsonl`, as the paper does;\n2. compare `nocot` vs `cot` vs `nocot_shuffled` on the same model.\n\n| | |\n|---|---|\n| **Archetype** | Single-turn, finite, deterministic — **no container, no Docker** |\n| **Harness** | Any; `null` for the pure protocol |\n| **Runtime** | No container (`NEEDS_CONTAINER` unset); run on a host `subprocess` runtime |\n| **Reward** | `correct` (weight 1.0): the paper's own substring grading, ported line-for-line |\n| **Data** | Fetched from the pinned upstream commit at first load; cached locally |\n\n## Conditions\n\n| Condition | Prompt | Rows | Upstream use |\n|---|---|---|---|\n| `cot` | asks for step-by-step reasoning, then the answer | 1,380 | graded accuracy |\n| `nocot` | demands the answer immediately, nothing else | 1,380 | graded accuracy — the paper's headline |\n| `nocot_shuffled` | the `nocot` prompts over a shuffled person → entity assignment | 6,900 | **loss-only control**; not graded upstream |\n\n`nocot_shuffled` is graded here with the same substring rule for convenience, but it is\nthe paper's control, not a third protocol: upstream lists the shuffled files under\n`eval_datasets` (loss) and never under `evaluations` (accuracy), and each of the 1,380\nquestions appears five times with five different gold answers. It is unsuitable as a\ntraining split.\n\n## Fidelity\n\nPreserved verbatim:\n\n- every row's own system + user prompts, unmodified;\n- upstream file order within each relation file, and a fixed catalogue order across files\n  (`ancient_cities` first), so `-n 60` without shuffling is the `ancient_cities` domain:\n  `continent`, `founding_century_bc`, `modern_country`, 20 rows each;\n- `correct`, the grading from `latent_reasoning/evaluate.py::grade`: any expected answer\n  appears **case-insensitively as a substring** of the generation.\n\nDeviations from upstream:\n\n- **No fine-tuning.** See *What this measures*.\n- **First-hop signals are a port addition.** Upstream leaves `answer_intermediate` empty\n  on every semi-synthetic test row (0 of 9,660 rows carry one) and therefore reports no\n  first-hop metric and no meaningful valid-CoT signal for these files. This port joins the\n  bridge entity from each domain's `train/first_hop.jsonl` by the invented person's name\n  across the three upstream question templates (1,380/1,380 rows join at the pinned\n  commit) and applies the paper's first-hop and valid-CoT rules against it:\n  `first_hop_mentioned` uses the same substring rule as `correct`; `valid_cot` is correct\n  ∧ bridge named ∧ bridge appears **before** the answer. An unjoinable row reports n/a\n  (−1.0), never a guess. In `nocot`, `correct` without `first_hop_mentioned` firing is the\n  paper's latent-success criterion, which is only meaningful for a model that knows the\n  first-hop facts.\n- **Decoding is not pinned.** Upstream decodes greedily (`do_sample=False`, temperature\n  0.0) with `max_new_tokens` 50 for `nocot` and 200 for `cot` on a local HF pipeline. The\n  repository config keeps temperature 0.0 but leaves the token budget uncapped\n  (`sampling.max_tokens = null`) because reasoning-model APIs spend hidden tokens; set\n  `[sampling] max_tokens` in your own config to reproduce the cap.\n- **`nocot_shuffled` is graded.** Upstream evaluates it by loss only (see *Conditions*).\n\n## Honest additions (separate from benchmark semantics)\n\nThe `nocot` instruction — *\"without any other words before or after\"* — is stricter than\nthe paper's substring grader, which pays for any superstring (\"The answer is Europe.\"\nscores 1.0). The mismatch is inherited from upstream, whose own `grade()` applies the same\nrule under the same prompt, and the prompt is kept verbatim for fidelity. Training on\n`correct` would reward ignoring the instruction, and the substring rule is also farmable\nby enumeration (\"Europe Asia Africa…\"). Fidelity wins — `correct` stays exactly as\npublished — but the port makes both hazards measurable and avoidable:\n\n- `answer_spam` (metric): how many **other** rows' answers for the same relation appear\n  in the generation, matched on token boundaries so a pool answer inside the gold (`8`\n  inside `18`) is not counted — nonzero while `correct` pays is the exploit happening.\n- `strict_answer` (reward, **weight 0.0**): the generation with runs of whitespace and\n  `.,!` collapsed to single spaces, lowercased, must equal the raw lowercased gold. The\n  gold is not normalised, so the one punctuated `nocot` gold (`parks`/`state`,\n  `U.S. Virgin Islands`) cannot score 1.0 under `strict_answer`; this is a known\n  limitation kept as-is so the measured run below stays reproducible, to be fixed by\n  normalising both sides in a future version. Not applicable to `cot`, whose prompt asks\n  for reasoning first.\n\nFor RL use, flip the weights from config without touching benchmark code:\n\n```toml\n[env.taskset.task.rewards]\ncorrect = { weight = 0.0 }\nstrict_answer = { weight = 1.0 }\n```\n\n## Rewards and metrics\n\n| Name | Kind (weight) | Definition |\n|---|---|---|\n| `correct` | reward (1.0) | Upstream `grade()`: the expected answer appears case-insensitively as a substring of the reply |\n| `strict_answer` | reward (0.0) | Reply with runs of whitespace and `.,!` collapsed to one space, lowercased, equals the raw lowercased gold. `nocot` only |\n| `first_hop_mentioned` | metric | Bridge entity appears (substring rule) anywhere in the reply; −1.0 when the row is unjoinable |\n| `valid_cot` | metric | `correct` ∧ bridge named ∧ bridge appears before the answer; −1.0 when unjoinable |\n| `answer_spam` | metric | Count of *other* pool answers for the relation found in the reply, on token boundaries |\n| `answered` | metric | 1.0 when the reply is non-empty. Separates no answer from a wrong one |\n\n## Arguments\n\nSet on the taskset with `--env.taskset.<flag>`, or under `[env.taskset]` in a config toml.\n\n| Flag | Default | Meaning |\n|---|---|---|\n| `--env.taskset.domains` | all 18 | Domains to load, in catalogue order |\n| `--env.taskset.relations` | `[]` (every relation) | Restrict to these relation names across the chosen domains |\n| `--env.taskset.condition` | `nocot` | `cot`, `nocot` or `nocot_shuffled` |\n| `--env.taskset.max-per-file` | `0` (all) | Cap on rows taken from each (domain, relation) file |\n\nThe taskset is finite; `-n` selects the first *n* rows in catalogue order unless the run\nshuffles.\n\n## Dependencies\n\n`verifiers>=0.3.1` and `requests`. The first load fetches the 69 relation files of the\nrequested condition plus the 18 `train/first_hop.jsonl` files from\n`raw.githubusercontent.com` at the pinned commit and caches them under\n`$TMPDIR/two_hop_latent_7e15bd1b`; later loads are offline. No API keys, secrets or\ncontainer.\n\n## Run\n\n```bash\n# Offline test suite, from the rl-environments repository\nuv run pytest tests/test_two_hop_latent.py -q\n\n# Model-free: load the live pinned data and check every row against the ported grader\nuv run validate two-hop-latent-v1 -n 40 --runtime.type subprocess\n\n# The measured run: configs/two-hop-latent.toml sets openai/gpt-5-mini, 60 tasks,\n# temperature 0.0, condition nocot and the null harness\nuv run eval @ configs/two-hop-latent.toml --env.agent.runtime.type subprocess --env.timeout.episode 480\n\n# One domain under the cot protocol\nuv run eval two-hop-latent-v1 --model <model-id> --env.agent.harness.id null \\\n  --env.agent.runtime.type subprocess --env.taskset.domains '[\"bridges\"]' --env.taskset.condition cot\n```\n\nThe runtime flag matters: resolved eval configs default `env.agent.runtime.type` to\n`prime`, and even the `null` harness launches its program in a runtime, so an unmodified\nrun provisions a Prime sandbox for every rollout although nothing here needs a container.\n`--env.timeout.episode 480` bounds a rollout stuck in provider retries.\n\n## Status\n\n| Check | Result |\n|---|---|\n| `uv run validate two-hop-latent-v1 -n 40 --runtime.type subprocess` | 40/40 valid against the live pinned data |\n| Offline test suite (pytest, in the repository) | Passes. Mirrors the upstream `grade()` semantics case by case (case-insensitive substring, superstrings pay, wrong and empty replies score 0); asserts that enumeration pays under `correct` but is visible in `answer_spam` and refused by `strict_answer`, that a pool answer inside the gold is not counted as spam, that an unjoinable row reports −1.0 rather than a guess, and that the cache is written atomically. One strict `xfail` tracks the `strict_answer` punctuated-gold limitation and flips when it is fixed |\n| Eval — `openai/gpt-5-mini` via Prime Inference, 2026-09-11 | `condition = nocot`, 60 tasks with `shuffle = false`, i.e. the first 60 catalogue rows (`ancient_cities`: `continent`, `founding_century_bc`, `modern_country`, 20 each), temperature 0.0, `null` harness, `subprocess` runtime: 60/60 `agent_completed`, 0 provider errors, ~9 min wall clock. `correct` 0.233 (14/60); `strict_answer` (weight 0.0) 0.200 (12/60); `first_hop_mentioned` 0.017; `valid_cot` 0.000; `answer_spam` 0.500. All 14 correct rows had `answer_spam` 0.0 and `first_hop_mentioned` 0.0; spam (mean 0.652) occurred only among the 46 incorrect rows; 2 of 14 correct rows fail strict equality |\n| Eval — `openai/gpt-5-nano`, 2026-09-12 | same 60 tasks and settings: 60/60 completed. `correct` 0.217 (13/60); `strict_answer` 0.150; `first_hop_mentioned` 0.033; `valid_cot` 0.000; `answer_spam` 0.467 |\n| Eval — `deepseek/deepseek-v4-flash`, 2026-09-12 | same 60 tasks and settings: 60/60 completed. `correct` 0.217 (13/60); `strict_answer` 0.183; `first_hop_mentioned` 0.017; `valid_cot` 0.000; `answer_spam` 0.250 |\n| Eval — `google/gemini-2.5-flash-lite`, 2026-09-12 | same 60 tasks and settings: 60/60 completed. `correct` 0.233 (14/60); `strict_answer` 0.217; `first_hop_mentioned` 0.017; `valid_cot` 0.000; `answer_spam` 0.333 |\n| Reading across the four models | The three 2026-09-12 rows report `answer_spam` under the token-boundary rule now in the code; the gpt-5-mini row's 0.500 is under the earlier substring rule (see the note below), so compare `answer_spam` across those rows only directionally. `correct` sits at 0.217–0.233 for every model, from the cheapest to the strongest tried. That flatness is what *What this measures* predicts for models that never saw the first-hop facts: the number is a prior over small closed answer sets, not latent two-hop reasoning. The port is doing its job — the signal it is built to detect only appears after the upstream fine-tune |\n| `cot` condition | Not yet run against a model |\n| `nocot_shuffled` condition | Not yet run against a model |\n| The other 17 domains | Not yet run against a model |\n| A model fine-tuned on `train/first_hop.jsonl` (the paper's protocol) | Not yet run |\n\nReading of the measured run: none of the 14 credited answers was farmed off the substring\nrule and none wrote the bridge entity, so the headline number is clean of the exploit.\ngpt-5-mini was not fine-tuned on the invented first-hop facts, however, so 14/60 should be\nread as a prior/guessing baseline on `ancient_cities`, not as evidence of latent two-hop\ncomposition (see *What this measures*).\n\nCaveats:\n\n- Prime Inference returned bursts of upstream 500s on gpt-5-mini during that session;\n  `--env.timeout.episode 480` was used to prevent hangs and is part of the command above.\n- `answer_spam` 0.500 was computed under the substring rule in force during the run. The\n  metric now matches on token boundaries; on `founding_century_bc` the old rule counted\n  pool answers contained inside the gold, so the measured value is not comparable to what\n  the current metric reports.\n- `answered` was added after that run; no measured value exists for it.\n\n## Not covered\n\nThe paper's frontier-model experiment (`real_facts_frontier_models`, the 42 MB\n`hopping_too_late` set), the fully-synthetic training splits and the fine-tuning half of\nthe semi-synthetic protocol are outside this port.\n","encoding":"utf-8","truncated":false,"total_bytes":13407},"status":null}