{"data":{"kind":"file","path":"README.md","version_id":"e01gph9arg54qzpbupln8wqf","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":4958,"modified_at":"2026-08-25T20:48:37.177000","content_hash":"32075bf9bbed1ad9450b8986c1eae40c3949a5ed8118cb76b76ff4f35123fa67"},"entries":[],"content":"# calendar-math\n\nSingle-turn **calendar arithmetic** for RLVR / evals on the [Prime Intellect Environments Hub](https://app.primeintellect.ai/dashboard/environments).\n\nHub: [devtechedge/calendar-math](https://app.primeintellect.ai/dashboard/environments/devtechedge/calendar-math) · Source: [github.com/devtechedge/calendar-math](https://github.com/devtechedge/calendar-math)\n\nThe model is given one of three question types, reasons, and puts a final answer in `<answer>` tags. The grader is pure `datetime` — no LLM-as-judge, no fuzzy string matching on the main reward.\n\n| Task | Example prompt | Gold answer |\n| --- | --- | --- |\n| `add_days` | What date is 1 day after 2024-02-28? | `2024-02-29` |\n| `days_between` | How many days after 2024-02-28 is 2024-03-01? | `2` |\n| `weekday` | What day of the week is 2024-02-29? | `Thursday` |\n\nThis is intentionally **not** reverse-text or word-count. Calendar reasoning is a documented LLM failure mode (leap years, century years, month lengths, weekday). The environment turns that into a dense, automatically-graded RL signal.\n\n## Why this design\n\n- **Verifiable.** Gold answers are produced by Python `datetime.date`. The same functions are the reference solver.\n- **Hard where it matters.** Eval always includes curated edge cases: 1900-02-28 (century, not leap), 2000-02-28 (century, leap), 2024-02-29, year boundaries.\n- **Not gameable by format alone.** Format is a 0.2 bonus. Exact match is the 1.0 term.\n- **Shaping, not noise.** Off-by-one dates / day-counts score 0.5 partial credit — models routinely confuse inclusive vs exclusive counting. Adjacent weekdays score 0.3.\n- **Configurable.** `num_train_examples`, `num_eval_examples`, `seed`, and an optional `task` pin.\n\n## Reward\n\n```\nreward = 1.0 * exact_match + 0.2 * format + 0.2 * partial_credit\n```\n\n| Term | 1.0 when | Notes |\n| --- | --- | --- |\n| `exact_match` | parsed `<answer>` equals gold | dates ISO, weekdays canonical English, counts decimal integers |\n| `format` | `<answer>...</answer>` present | extra prose outside the tags is ignored |\n| `partial_credit` | near-miss as above | 0 when exact match already fired, so a perfect answer is **1.2** not 1.4 |\n\n## Eval\n\n`vf-eval` on the 15 curated edge cases (`num_eval_examples=15`, 1 rollout):\n\n| Policy | avg reward | exact | format | partial |\n| --- | --- | --- | --- | --- |\n| Gold datetime solver (ceiling) | **1.200** | 1.000 | 1.000 | 0.000 |\n| Naive calendar (year%4 leaps, inclusive counts) | **0.768** | 0.533 | 1.000 | 0.173 |\n\nThe gold policy is a harness check: install, `load_environment`, rollouts, and the rubric all fire. The naive policy is a discrimination check: century non-leaps and inclusive day-counts do not rubber-stamp 1.2.\n\nAgainst an API model (needs `OPENAI_API_KEY` or `--provider prime`):\n\n```bash\nuv run vf-eval calendar-math -n 20 -r 1 -m gpt-4.1-mini\n```\n\n## Installation\n\n```bash\nuv pip install -e .\npython -m pytest tests/test_calendar_math.py -q\n```\n\nFrom the Hub:\n\n```bash\nprime env install devtechedge/calendar-math\n```\n\n```python\nimport verifiers as vf\n\nenv = vf.load_environment(\"calendar-math\")\n```\n\n## `load_environment` arguments\n\n| Arg | Default | Meaning |\n| --- | --- | --- |\n| `num_train_examples` | `500` | train split size |\n| `num_eval_examples` | `100` | eval split size (edge cases prepended) |\n| `seed` | `42` | train RNG; eval uses `seed + 1` |\n| `task` | `None` | `\"add_days\"` \\| `\"days_between\"` \\| `\"weekday\"` \\| mixed |\n\n```bash\nuv run vf-eval calendar-math -n 20\nuv run vf-eval calendar-math -a '{\"task\": \"weekday\", \"num_eval_examples\": 40}'\n```\n\nRequires `verifiers>=0.1.14`. Dataset rows use `task_type` (not `task`): current verifiers treat `info[\"task\"]` as a nested rollout payload.\n\n## Gold solution\n\nDataset construction **is** the gold solver. For a row `info`:\n\n```python\nfrom datetime import date, timedelta\n\nWEEKDAYS = [\"Monday\", \"Tuesday\", \"Wednesday\", \"Thursday\",\n            \"Friday\", \"Saturday\", \"Sunday\"]\n\ndef gold(info):\n    if info[\"task_type\"] == \"add_days\":\n        return (date.fromisoformat(info[\"start\"]) + timedelta(days=info[\"n\"])).isoformat()\n    if info[\"task_type\"] == \"days_between\":\n        a = date.fromisoformat(info[\"start\"])\n        b = date.fromisoformat(info[\"end\"])\n        return str((b - a).days)          # midnights that pass; same day = 0\n    return WEEKDAYS[date.fromisoformat(info[\"start\"]).weekday()]\n```\n\n`tests/test_calendar_math.py` asserts the solver against a hand-checked fixture list (leap years, century years, negative offsets, same-day diffs).\n\n## Files\n\n```\ncalendar_math.py              # generator, gold, grader, load_environment\npyproject.toml\nREADME.md\nLICENSE\ntests/test_calendar_math.py\n```\n\n## What this is not\n\n- Not a wrap of GSM8K or any public dataset.\n- Not LLM-judged.\n- Not multi-turn / tool-using. Those are the right shape for a **second** environment (meeting-conflict scheduler, timezone conversion with a tz database, etc.).\n\n## License\n\nMIT\n","encoding":"utf-8","truncated":false,"total_bytes":4958},"status":null}