{"data":{"kind":"file","path":"README.md","version_id":"p6duqghp3uzl19k8rpmrmxqa","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":4978,"modified_at":"2026-08-04T11:54:28.528000","content_hash":"c0821c18b142780506d9ddbac32d758300df81a43fb8aaee199ebc35d9b44fde"},"entries":[],"content":"# rfc3339-strict\n\nStrict RFC 3339 `date-time` conformance and UTC canonicalisation, with a grader built to be\nhard to fool.\n\n### Overview\n- **Environment ID**: `rfc3339-strict`\n- **Short description**: Judge whether a timestamp conforms to RFC 3339 §5.6, canonicalise it\n  to UTC, and name the first violated rule — graded all-or-nothing and type-aware.\n- **Tags**: `parsing`, `spec-conformance`, `datetime`, `eval`, `single-turn`\n\n### Why this exists\n\nThe interesting part of a timestamp eval is not the timestamps, it is the **grader**. Date\nhandling is where permissive checks quietly score wrong answers as right:\n\n- `v == True` accepts the integer `1`, because in Python `True == 1`.\n- `str(a) == str(b)` accepts `\"1\"` for `1`.\n- A case-insensitive compare hides a canonicalisation bug.\n- A format-conformance reward creates a gradient a model can climb without ever getting a\n  timestamp right.\n\nEvery one of those is planted as a defect in `selftest.py` and must be caught.\n\n### Task\n\nSingle-turn. The model receives the full spec as a system prompt and one candidate string,\nand must reply with exactly one JSON object:\n\n```json\n{\"valid\": true, \"utc\": \"2025-12-31T18:30:00Z\", \"violation\": null}\n```\n\n- `valid` — must be a JSON **boolean**.\n- `utc` — canonical `YYYY-MM-DDTHH:MM:SS[.frac]Z`, trailing fractional zeros stripped;\n  `null` when invalid.\n- `violation` — the **first** applicable code in a fixed precedence order\n  (`shape` → `month` → `day` → `hour` → `minute` → `second` → `offset`); `null` when valid.\n\nEverything the grader enforces is stated in the prompt. A rule the grader applies but the\nprompt withholds is not difficulty, it is a trick question.\n\n### Dataset\n\n**50 constructed cases** — 20 valid, 30 invalid — not sampled. Boundaries are built\ndeliberately: leap days (`2024-02-29` valid, `1900-02-29` not, `2000-02-29` valid), leap\nseconds (`23:59:60` is legal per the grammar), offset extremes (`+23:59`, `-00:00`),\nfractional-second trimming (`.500` → `.5`, `.000` → dropped), year bounds (`0001`, `9999`),\nlowercase `t`/`z`, and shape traps (space separator, unpadded fields, `+0530`, trailing\nwhitespace). Two cases have several fields wrong at once, to pin the precedence rule.\n\n### Rubric\n\n| Function | Weight | Meaning |\n| --- | --- | --- |\n| `exact_match` | **1.0** | All three fields correct. Type-aware, byte-exact, all-or-nothing. |\n| `parses_as_contract` | **0.0** | Diagnostic only — did the reply parse to the 3-key contract? |\n\n`parses_as_contract` is deliberately weighted **zero**. A well-formed-but-wrong answer scoring\nabove zero is how a format reward becomes the thing being optimised.\n\n### Ground truth is double-sourced\n\nEvery case carries a **hand-written** expected answer. `_reference_judge()` is a second,\nindependent implementation of the same spec. Neither is authoritative — `selftest.py` asserts\nthey agree on all 50 cases and fails loudly on disagreement. A generator that emits both an\ninput and its own answer cannot detect its own bug; two sources can.\n\n### Control\n\n```bash\nuv run python selftest.py\n```\n\nCurrent state — **89 assertions, 0 failures, 9 planted defects, all caught**:\n\n```\noracle (perfect answers)          1.0000\nbest constant answer              0.2400   ← always {\"valid\":false,...,\"violation\":\"shape\"}\n```\n\nMeasured constant-answer baselines, so the reward-hacking floor is stated rather than assumed:\n\n| Constant strategy | Score |\n| --- | ---: |\n| always `invalid` / `shape` | 0.2400 |\n| always `invalid` / `day` | 0.1600 |\n| always `valid`, fixed UTC | 0.1000 |\n| always `invalid` / `month` | 0.0600 |\n| always `invalid` / `hour` | 0.0600 |\n| always `invalid` / `offset` | 0.0400 |\n| always `invalid` / `minute` | 0.0200 |\n| always `invalid` / `second` | 0.0200 |\n\nNo constant strategy clears 0.25. The `violation` field is what makes this true: without it,\n\"always invalid\" would score 0.60.\n\nPlanted defects, each proven to be caught: type-blind boolean compare, case-insensitive UTC\ncompare, key-set contract check removed, `utc`-must-be-null check removed, violation code never\nchecked, valid-branch ignores `utc`, grader always returns 1.0, and two mutations of the\nreference implementation (leap-year century rule, leap-second rejection).\n\n> One mutation is deliberately **not** listed: changing `is not` to `!=` on the boolean compare\n> is an *equivalent mutant* — with the `isinstance(v, bool)` guard in place it changes nothing\n> observable. Listing it would give a permanently red assertion that says nothing about grader\n> quality.\n\n### Quickstart\n\n```bash\nprime eval run rfc3339-strict\nprime eval run rfc3339-strict -m openai/gpt-4.1-mini -n 50 -r 3\n```\n\n### Metrics\n\n| Metric | Meaning |\n| --- | --- |\n| `reward` | Fraction of cases fully correct across all three fields (0.0–1.0) |\n| `exact_match` | Same as `reward`; the only weighted criterion |\n| `parses_as_contract` | Diagnostic: fraction of replies matching the 3-key contract |\n","encoding":"utf-8","truncated":false,"total_bytes":4978},"status":null}