{"data":{"kind":"file","path":"README.md","version_id":"p6oin7ibv22ud04n341cnqsx","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5326,"modified_at":"2026-10-23T02:41:43.802000","content_hash":"8afbee4a1e97e57a7086ba109c329432429b9f86b70d748266b6643e22f52595"},"entries":[],"content":"# cron-reasoning\n\nWrite and execute cron schedules. Tasks are procedurally generated, so the supply is\nunlimited and seeded, and scoring is exact: no LLM judge, no sandbox, no network.\n\n## Tasks\n\n- **write**: an English schedule (\"Every 15 minutes between 07:00 and 11:59 on weekdays\n  (Monday through Friday).\") must become a 5-field cron expression. Any expression that\n  fires at exactly the same minutes as the reference scores 1. `*/30 * * * *`,\n  `0,30 * * * *` and `0-59/30 * * * *` are all accepted.\n- **read**: a cron expression and a start time must become the next N run times (default 3),\n  strictly after the start.\n\nSchedules combine minute steps and lists, hour lists, ranges and steps, day-of-month lists\nand ranges, weekday lists and names, month lists, ranges and quarterly steps. Level 3 also\nuses tasks where both day fields are restricted, which fire when **either** field matches\n(Vixie cron's OR rule). Models often get this wrong.\n\n## Scoring\n\n| Name | Weight | Meaning |\n| --- | --- | --- |\n| `correct` | 1.0 | write: 1 if the expression in the last `<cron>` block is equivalent to the reference, else 0. read: share of run times in the last `<times>` block that are correct and in order; extra lines count against the score. |\n| `formatted` | metric | 1 if the expected answer tag is present. |\n\nEquivalence is decided exactly, not by sampling. Two schedules are equivalent iff their\nminute sets, hour sets and day rules match. The day rule is compared over every\n(month, day-of-month, weekday) triple, and that comparison is exact because every\nGregorian date (Feb 29 included) falls on every weekday within the 400-year cycle. The\ntests check that claim by brute force.\n\nSemantics follow Vixie cron / cronie: `N/STEP` means `N-max/STEP`, 0 and 7 are both\nSunday, `JAN`-`DEC` and `SUN`-`SAT` names and `@daily`-style macros are accepted, and a day\nfield starting with `*` (including `*/2`) switches the day rule to AND. Non-standard\nextensions (`L`, `W`, `#`, `?`, a seconds field) are rejected. There are no time zones or DST\nchanges.\n\n## Configuration\n\n| Flag | Default | Meaning |\n| --- | --- | --- |\n| `--env.taskset.mode` | `mixed` | `write`, `read` or `mixed` |\n| `--env.taskset.num-tasks` | `1000` | Number of generated tasks |\n| `--env.taskset.num-times` | `3` | Run times asked for in `read` tasks |\n| `--env.taskset.seed` | `0` | Generator seed; the same seed always gives the same tasks |\n\n## Usage\n\n```bash\nuv pip install -e .\nuv run vf-validate cron-reasoning -n 50 --runtime.type subprocess\nuv run vf-eval cron-reasoning --env.agent.harness.id null --env.agent.runtime.type subprocess -n 20 -r 3\n```\n\n## Validation\n\n- `vf-validate` (gold + noop) on 50 tasks: reference answers score 1.0 and an empty reply\n  scores 0.0 on all 50.\n- `pytest tests`: the 400-year calendar claim checked by brute force,\n  equivalence and non-equivalence cases including the OR/AND day rule, invalid syntax,\n  `next_times` against minute-by-minute brute force and against `croniter` on 300+\n  generated schedules, scorer edge cases, and gold/noop on 300 tasks.\n\n## Results\n\n50 tasks (seed 0, mixed mode) × 3 rollouts per model, `null` harness, `max_tokens=16384`,\nPrime Inference. \"Solved\" means reward 1.0. Total cost: about $0.55.\n\n| Model | All | write | read | read, OR-rule tasks | Median output tokens | Tasks with mixed rewards |\n| --- | --- | --- | --- | --- | --- | --- |\n| DeepSeek V4.1 Flash | 93% | 91% | 95% | 6/6 | 192 | 9/50 |\n| Qwen3.5-9B | 80% | 96% | 57% | 0/6 | 4,525 | 15/50 |\n\nQwen3.5-9B had 4 rollouts fail with provider 503 errors. Those are excluded, not counted\nas 0.\n\nWhat the failures show (every failed rollout was read by hand):\n\n- **The two task types test different skills.** The small model translates English into\n  cron almost perfectly (96%) but executes cron poorly (57%). The larger model does both.\n- **Executing cron is where the small model breaks.** 18 of its 61 read rollouts used all\n  16k tokens doing calendar arithmetic and never answered. The rest went wrong at the\n  start (the first `*/5` run after 15:48 is 15:50, not 16:00), ignored the hour field, or\n  applied the day-of-month/day-of-week OR rule as AND: for `*/10 * 20 6-9 0` it waited\n  for June 20 and skipped Sunday June 5.\n- **The strong model's errors are mostly formatting.** 8 of DeepSeek's 11 failures had the\n  right expression but the wrong tag (`<cronexpr>`, `<cro>…</cron>`, a made-up tool call).\n  Its two real mistakes swapped fields: it read day-of-month `9,17,20` as hours.\n- **Training signal.** 9–15 of 50 tasks have rollouts that disagree, which is the variance\n  GRPO-style training needs. Read tasks give a 9B model a clear target.\n- **Difficulty levels don't separate strong models.** Level 3 was no harder than level 1\n  for DeepSeek, so the current levels mostly add variety. Harder variants (longer horizons\n  across leap days, more run times per task) are the natural next step.\n\nScorer check: version 0.1.0 only accepted `<cron>` exactly. 4 DeepSeek rollouts wrote\n`< cron>` with a space, so 0.1.1 tolerates whitespace inside the tag brackets. The results\nabove were rescored offline with `vf-replay` on the same rollouts.\n\n## Changelog\n\n- 2026-10-22 (0.1.1): Accept whitespace inside answer tags (`< cron>`); add eval results.\n- 2026-10-22 (0.1.0): Initial v1 taskset.\n","encoding":"utf-8","truncated":false,"total_bytes":5326},"status":null}