{"data":{"kind":"file","path":"README.md","version_id":"l3y4975oc99e6u8xlpl61dx9","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2944,"modified_at":"2026-08-15T14:51:49.685000","content_hash":"196ce1b94a2823d79e6b29cda2d69b487e7c91a05a887366b983423a5354dc19"},"entries":[],"content":"# specdecode-rl\n\nA stateful environment for adaptive speculative decoding.\n\nAt each verification round, the agent chooses `k ∈ {0, 1, 2, 4, 8, 16}`. Larger `k` can commit more tokens, but drafting and verification both cost time. `k=0` skips speculation and runs the target model normally.\n\nThe goal is to minimize latency without changing the target model's output.\n\n## Task\n\nThe agent receives generation progress, previous draft outcomes, acceptance history, draft confidence, and latency so far. It responds with one tool call:\n\n```json\n{\"k\": 4}\n```\n\nThe main reward is:\n\n```text\n1 - policy_latency / target_only_latency\n```\n\nInvalid actions are penalized, and output must match the target trace exactly.\n\n## Data\n\nBy default, the environment uses deterministic synthetic episodes covering prose, code, reasoning, and low draft/target agreement.\n\nIt can also replay JSONL traces collected from real model pairs. The project includes traces from 50 `nvidia/SPEED-Bench` prompts profiled on an A100-SXM4-80GB:\n\n```text\nTarget: Qwen/Qwen3-8B\nDraft:  Qwen/Qwen3-1.7B\nDtype:  bfloat16\nOutput: 48 tokens\n```\n\nPass a trace file through `trace_path` to use real replay instead of synthetic episodes.\n\n## Experiments\n\nResults on a shared 35/5/10 train/validation/test split, averaged over three seeds for learned policies:\n\n| Policy | Mean test speedup |\n|---|---:|\n| Target-only | 1.0000× |\n| REINFORCE | 1.0056× ± 0.0145 |\n| Fixed `k=2` | 1.0192× |\n| **Oracle imitation** | **1.0393× ± 0.0135** |\n| Hindsight oracle | 1.2307× |\n\nREINFORCE mostly converged to near-fixed actions. Oracle imitation learned to speculate selectively within an episode and performed better on this split.\n\nRemoving acceptance and confidence history reduced mean speedup from `1.0393×` to `1.0283×`. Removing explicit token-position features had little effect (`1.0362×`).\n\nW&B runs: https://wandb.ai/semioz/specdecode-rl\n\n## Quickstart\n\nSynthetic episodes:\n\n```bash\nprime eval run specdecode-rl -m openai/gpt-4o-mini -n 5 -r 3\n```\n\nReal trace replay:\n\n```bash\nprime eval run specdecode-rl -m openai/gpt-4o-mini \\\n  -n 10 -r 3 \\\n  -a '{\"trace_path\": \"traces/speed-qwen.jsonl\"}'\n```\n\nArguments:\n\n| Field | Default | Description |\n|---|---:|---|\n| `count` | `40` | Training episodes loaded from the trace file |\n| `eval_count` | `0` | Held-out evaluation episodes |\n| `seed` | `42` | Trace shuffle seed |\n| `output_length` | `48` | Synthetic episode length |\n| `trace_path` | `None` | Optional real JSONL trace file |\n| `regimes` | all | Synthetic regimes to include |\n\n## Scope\n\nThis is deterministic offline trace replay, not a live vLLM or SGLang integration. The real traces use batch size 1 with no serving queue or concurrency. The dataset is small, and the test traces had appeared in aggregate exploration before being held out from training, so the reported speedups are not production serving claims.\n\nSource: https://github.com/semioz/specdecode-rl\n","encoding":"utf-8","truncated":false,"total_bytes":2944},"status":null}