{"data":{"kind":"file","path":"README.md","version_id":"vub51oogn2nrcfnurcoqgfi5","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6250,"modified_at":"2026-08-25T21:03:55.732000","content_hash":"4cc7e33bd3a492599c7439af4198b7c621da5b7b89381c1f20a8b9448767aa35"},"entries":[],"content":"# meeting-slot\r\n\r\nMulti-turn **tool-using meeting scheduler** for RLVR / evals on the [Prime Intellect Environments Hub](https://app.primeintellect.ai/dashboard/environments).\r\n\r\nHub: [devtechedge/meeting-slot](https://app.primeintellect.ai/dashboard/environments/devtechedge/meeting-slot) · Source: [github.com/devtechedge/meeting-slot](https://github.com/devtechedge/meeting-slot)\r\n\r\nA sequel to [calendar-math](https://github.com/devtechedge/calendar-math): same calendar domain, but the model has to **query tools** instead of reading the calendar out of the prompt.\r\n\r\nThe task is to find the **earliest valid UTC start** that works for every attendee. Busy intervals, working hours, and timezones are hidden. The final answer goes in `<answer>` tags as a UTC ISO-8601 timestamp, e.g. `2024-03-11T15:00:00Z`. If no slot exists, `NONE`.\r\n\r\nThe grader is a UTC sweep-line over `zoneinfo` — no LLM-as-judge, no fuzzy string match on the main reward.\r\n\r\n| Tool | Returns |\r\n| --- | --- |\r\n| `list_attendees()` | JSON names |\r\n| `get_timezone(name)` | IANA timezone |\r\n| `get_working_hours(name)` | local `HH:MM` hours + weekdays (`0=Monday`) |\r\n| `get_busy(name, date)` | local half-open busy intervals for that **local** date |\r\n\r\n## Why this design\r\n\r\n- **Verifiable.** Gold answers are produced by interval intersection in UTC. The same functions are the reference solver.\r\n- **Tool use is load-bearing.** The world is not in the prompt. A model that does not call tools cannot solve the task.\r\n- **Hard where it matters.** Eval always includes curated edge cases: US/EU DST, NY↔Kolkata no-overlap, Friday 16:30 vs Monday 09:00, 30 vs 60 minute gaps, four-person summer overlap, inclusive-busy traps.\r\n- **Not gameable by format alone.** Format is a 0.2 bonus. Exact match is the 1.0 term.\r\n- **Shaping, not noise.** A conflict-free in-hours start that is not the earliest scores 0.5 partial credit. Invalid overlap / wrong duration / outside hours score 0 on the main terms.\r\n- **Configurable.** `num_train_examples`, `num_eval_examples`, `seed`, `max_turns`, optional `difficulty` pin.\r\n\r\n## Reward\r\n\r\n```\r\nreward = 1.0 * exact_match + 0.2 * format + 0.2 * partial_credit\r\n```\r\n\r\n| Term | 1.0 when | Notes |\r\n| --- | --- | --- |\r\n| `exact_match` | parsed `<answer>` equals gold | UTC ISO with `Z`, or `NONE` |\r\n| `format` | `<answer>...</answer>` present | extra prose outside the tags is ignored |\r\n| `partial_credit` | valid-but-not-earliest | 0.5; 0 when exact match already fired, so a perfect answer is **1.2** not 1.4 |\r\n\r\nIntervals are half-open `[start, end)`. A busy block ending at 11:00 means 11:00 is free.\r\n\r\n## Eval\r\n\r\n15 curated edge cases (`num_eval_examples=15`, 1 rollout each).\r\n\r\n| Policy | avg reward | exact | format | partial |\r\n| --- | --- | --- | --- | --- |\r\n| Gold sweep-line via tools (ceiling) | **1.200** | 1.000 | 1.000 | 0.000 |\r\n| Naive (ignore TZ; busy end inclusive) | **0.293** | 0.067 | 1.000 | 0.133 |\r\n| `minimax/minimax-m2.7` (OpenRouter, T=0, 2048 tok) | **0.993** | 0.800 | 0.933 | 0.033 |\r\n\r\nThe gold policy is a harness check: tools leak enough to rebuild the world, and the rubric fires 1.2. The naive policy is a discrimination check: DST offsets, NY/Kolkata non-overlap, and inclusive busy do not rubber-stamp 1.2. Naive is exact on only the fully-booked `NONE` row.\r\n\r\nMiniMax is exact on **12/15**. The three misses are the ones the env is supposed to catch:\r\n\r\n- `sydney_ny_none` — format only (0.2). Claimed a 13:00Z overlap between Sydney and New York that does not exist.\r\n- `dst_eu_monday` — valid-not-earliest (0.3). Answered 10:00Z after the EU spring-forward; gold is 08:00Z.\r\n- `no_slot_fully_booked` — 0.0. Truncated before `</answer>` at 2048 tokens.\r\n\r\n```bash\r\nuv run vf-eval meeting-slot -n 15 -r 1 -p openrouter \\\r\n  -m minimax/minimax-m2.7 --max-tokens 2048 \\\r\n  --temperature 0 --max-concurrent 1 --disable-tui --disable-env-server\r\n```\r\n\r\n`--max-concurrent 1` keeps OpenRouter in-flight budget from aborting mid-rollout. Budget ≥2048 max tokens. Reasoning models spend the window on tool calls before `</answer>`; truncation looks like a grader bug.\r\n\r\n## Installation\r\n\r\n```bash\r\nuv pip install -e .\r\npython -m pytest tests/test_meeting_slot.py -q\r\n```\r\n\r\nFrom the Hub:\r\n\r\n```bash\r\nprime env install devtechedge/meeting-slot\r\n```\r\n\r\n```python\r\nimport verifiers as vf\r\n\r\nenv = vf.load_environment(\"meeting-slot\")\r\n```\r\n\r\nRequires `verifiers>=0.1.14,<0.2`.\r\n\r\n## `load_environment` arguments\r\n\r\n| Arg | Default | Meaning |\r\n| --- | --- | --- |\r\n| `num_train_examples` | `500` | train split size |\r\n| `num_eval_examples` | `100` | eval split size (edge cases prepended) |\r\n| `seed` | `42` | train RNG; eval uses `seed + 1` |\r\n| `max_turns` | `40` | tool-call turns before stop |\r\n| `difficulty` | `None` | `\"easy\"` \\| `\"medium\"` \\| `\"hard\"` \\| mixed |\r\n\r\n```bash\r\nuv run vf-eval meeting-slot -n 20\r\nuv run vf-eval meeting-slot -a '{\"difficulty\": \"hard\", \"num_eval_examples\": 40}'\r\n```\r\n\r\nDataset rows never use a column named `task`. Verifiers ≥0.1 treats `info[\"task\"]` as a nested rollout payload.\r\n\r\n## Gold solution\r\n\r\nDataset construction **is** the gold solver. For each attendee, working hours minus busy are converted to UTC with `zoneinfo` (both DST folds, skipping spring-forward gaps). The UTC free intervals are intersected; the earliest start `s` with `s + duration` inside the intersection and inside the search window is gold.\r\n\r\n```python\r\nfrom meeting_slot import gold_earliest, solve_from_tools\r\n\r\ngold_earliest(world, duration_minutes, window_start, window_days)\r\n# identical, but only using the four public tools:\r\nsolve_from_tools(world, duration_minutes, window_start, window_days)\r\n```\r\n\r\n`tests/test_meeting_slot.py` asserts the solver against a hand-checked fixture list (US/EU DST, 30 vs 60 minute gaps, NY–Kolkata NONE, inclusive busy).\r\n\r\n## Files\r\n\r\n```\r\nmeeting_slot.py                 # generator, gold, tools, grader, load_environment\r\npyproject.toml\r\nREADME.md\r\nLICENSE\r\ntests/test_meeting_slot.py\r\n```\r\n\r\n## What this is not\r\n\r\n- Not a wrap of a public calendar dataset.\r\n- Not LLM-judged.\r\n- Not single-turn. The calendars are behind tools on purpose.\r\n- Not a dump of the whole calendar into the prompt.\r\n\r\n## License\r\n\r\nMIT\r\n","encoding":"utf-8","truncated":false,"total_bytes":6250},"status":null}