{"data":{"kind":"file","path":"README.md","version_id":"ebu91lcr5kyioeh0b68msm9g","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6015,"modified_at":"2026-08-30T10:33:55.350000","content_hash":"510c436538c9ec599e537bcdb5220df6e00f88903cdbee5f35519867b4d4ddb8"},"entries":[],"content":"# captionsmith\n\n`captionsmith` is a deterministic [Verifiers](https://github.com/PrimeIntellect-ai/verifiers)\nv1 taskset for evaluating and training language models to convert word-timed ASR\ntranscripts into valid, readable SRT subtitles.\n\nThe model receives words with start/end seconds and must decide:\n\n- which words belong in each subtitle cue;\n- where the one- or two-line break belongs;\n- the cue's millisecond start and end timestamps;\n- how to satisfy timing, reading-speed, line-length, and gap constraints together.\n\nNo customer transcripts or FrameOS production data are included. Tasks are\ngenerated procedurally from a fixed seed and are feasibility-checked before they\nare emitted.\n\nInstall the public Hub release:\n\n```bash\nprime env install thesatvik/captionsmith@0.1.1\n```\n\n## Why this task\n\nSubtitle layout is a useful structured-output benchmark because a valid answer\nrequires exact text preservation, temporal arithmetic, global segmentation, and\nmultiple interacting constraints. It is also directly usable for reinforcement\nlearning: every reward is deterministic and requires no LLM judge or external API.\n\nThe default profile is based on public Netflix timed-text guidance:\n\n- [42 characters per line and at most two lines](https://partnerhelp.netflixstudios.com/hc/en-us/articles/30806198616339-English-UK-Timed-Text-Style-Guide);\n- [17 characters per second for English subtitle templates](https://partnerhelp.netflixstudios.com/hc/en-us/articles/219375728-Timed-Text-Style-Guide-Subtitle-Templates), counting [spaces and punctuation](https://partnerhelp.netflixstudios.com/hc/en-us/articles/115001352212-How-is-reading-speed-measured-Do-punctuation-and-spaces-count);\n- a two-frame gap at 24 fps (`0.083s`) and cue durations within a conservative\n  `1–6s` subset of Netflix's [timing guidance](https://partnerhelp.netflixstudios.com/hc/en-us/articles/360051554394-Timed-Text-Style-Guide-Subtitle-Timing-Guidelines).\n\nThis environment is inspired by those public constraints; it is not affiliated\nwith Netflix and does not claim to reproduce Netflix's complete delivery spec.\n\n## Scoring\n\nA malformed or overlapping SRT file cannot be aligned and receives zero on all\nfour rewards. Valid output receives a weighted sum in `[0, 1]`:\n\n| Reward | Weight | What it measures |\n|---|---:|---|\n| `coverage` | 0.25 | LCS token F1: every source word retained in order, with no invented words |\n| `timing` | 0.25 | Cue starts/ends aligned to the words assigned to that cue |\n| `constraints` | 0.30 | Lines, characters, duration, reading speed, and inter-cue gaps |\n| `segmentation` | 0.20 | Boundaries at pauses/sentence ends/forced splits; no dangling connector words |\n\n`parse_ok`, `cue_count`, and `max_cps` are recorded as diagnostic metrics and do\nnot affect reward.\n\nThe source/output alignment uses longest-common-subsequence matching, so repeated\nwords, isolated omissions, and invented tokens are handled deterministically.\n\n## Development\n\nFrom a checkout of this repository:\n\n```bash\nuv sync\nuv run eval captionsmith --dry-run -n 3\nuv run validate captionsmith --only-gold -n 50 --runtime.type subprocess\n```\n\nRun a local endpoint-backed evaluation with a tool-free harness:\n\n```bash\nuv run eval captionsmith \\\n  -m openai/gpt-oss-20b -n 5 -r 1 \\\n  --env.agent.harness.id null \\\n  --env.agent.runtime.type subprocess \\\n  --client.base-url https://api.groq.com/openai/v1 \\\n  --client.api-key-var GROQ_API_KEY \\\n  --sampling.reasoning-effort low \\\n  --sampling.temperature 0.2 \\\n  --sampling.max-tokens 1000\n```\n\n## Configuration\n\nThe taskset is infinite and deterministic for a fixed `seed`; every evaluation\nmust set `-n`. Important overrides include:\n\n```text\n--env.taskset.seed\n--env.taskset.min-words / --env.taskset.max-words\n--env.taskset.disfluency-rate\n--env.taskset.punctuation-ratio\n--env.taskset.max-chars-per-line\n--env.taskset.max-lines\n--env.taskset.cps-cap\n--env.taskset.min-gap-s\n--env.taskset.min-cue-s / --env.taskset.max-cue-s\n```\n\nIf a custom constraint profile cannot produce a feasible task after 2,000\nattempts, loading fails with an actionable error instead of hanging forever.\n\n## Validation evidence\n\n- 50/50 generated tasks pass the native model-free `validate` command.\n- The built wheel also passes 10/10 gold tasks in a clean environment using the\n  released `verifiers==0.3.1`, independently of the development monorepo.\n- The published Hub artifact independently installs as\n  `thesatvik/captionsmith@0.1.0` and passes another 10/10 gold validation run.\n\nThe package accepts Prime Hosted Training's bundled native-v1 development build\n(`verifiers==0.2.2.dev6`) but clean installations resolve the current stable\n`verifiers==0.3.1`. The upper bound prevents unreviewed development releases\nfrom being selected automatically.\n- The hidden reference solution scores `1.0` on every reward for every validated task.\n- Malformed SRT scores `0.0` across all rewards.\n- Missing or invented words reduce token coverage; shifted timings reduce timing reward.\n- A one-task smoke run with `openai/gpt-oss-20b` (Groq, low reasoning effort,\n  temperature `0.2`) scored `0.80`: coverage `1.00`, segmentation `1.00`,\n  constraints `0.75`, timing `0.50`. This is a smoke result, not a model benchmark.\n- In a later five-task paced run, four provider calls returned: three parseable\n  answers scored `0.68`, `0.73`, and `0.87`; one answer spent its 1,000-token\n  budget on reasoning and emitted no SRT, scoring `0.00`. The fifth call received\n  an HTTP 429 at Groq's 8,000-token-per-minute free-tier limit.\n\nFull multi-model calibration should be performed with Prime-hosted inference or\nan endpoint with enough rate-limit headroom for the recommended 5 tasks × 3 rollouts.\n\n## Source and license\n\nHub: [thesatvik/captionsmith](https://app.primeintellect.ai/dashboard/environments/thesatvik/captionsmith) ·\nSource: [github.com/theSatvik/captionsmith](https://github.com/theSatvik/captionsmith) ·\nAuthor: [@theSatvik](https://github.com/theSatvik)\n\nMIT © Satvik Shrivas\n","encoding":"utf-8","truncated":false,"total_bytes":6015},"status":null}