{"data":{"kind":"file","path":"README.md","version_id":"oqexa8s239xgpcqryixsdb5u","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":7591,"modified_at":"2026-09-06T02:04:04.890000","content_hash":"e192bd93ae9843bd7ea0781911e1a9fd208d65b3991a2c4663ca210746b6bcaa"},"entries":[],"content":"# fleet-lock\n\nImplement the coordination registry that several agents share when they all commit\nto **one** git working tree: path claims with glob overlap, and a single global\ndeploy lock.\n\nThe model works in a container as a coding agent: it writes `registry.py` defining a\n`Registry` class, and is told to write its own scenarios and run them before\nfinishing. A scripted multi-session trace is then replayed against whatever it left\nbehind, and every decision is compared with ground truth - scoring is exact rather\nthan judged.\n\n| metric | meaning |\n|---|---|\n| `wrote_file` | did the agent create `registry.py` at all |\n| `safety` | of the operations that MUST be refused, the fraction refused |\n| `liveness` | of the operations that MUST be allowed, the fraction allowed |\n| **reward** | **`safety x liveness`** |\n\n## Why the reward is a product\n\nCoordination is the problem where **both trivial answers are exactly half right.**\nA registry that refuses everything is perfectly safe and deadlocks the fleet. One\nthat allows everything is perfectly live and is simply the absence of a lock. A\nmean rates both at 0.50 and cannot tell you which half you got. A product rates\nboth at zero.\n\n```\nstrategy               safety  liveness   product   (mean)\n-----------------------------------------------------------\nallow_everything         0.00      1.00      0.00     0.50\nrefuse_everything        1.00      0.00      0.00     0.50\ncrashes                  0.00      0.00      0.00     0.00\nno_class                 0.00      0.00      0.00     0.00\nfnmatch_glob             1.00      0.89      0.89     0.95\nexpired_is_free          0.92      0.96      0.88     0.94\nreference                1.00      1.00      1.00     1.00\n```\n\n## The two traps, and why they are planted rather than sampled\n\nA randomly drawn trace hits the interesting rules rarely, so an environment built\non one mostly measures luck. Every trace here has both traps spliced in as an\nordered block:\n\n**1. An EXPIRED lock is not FREE.** `lock_take` against an expired lock must be\n*refused*; `lock_steal` is the way through. An expired lock means someone's deploy\ndied holding it — treating that as an empty road is how two deploys end up running\nat once.\n\n**3. One lock across all lanes** (`lanes` tier only). `lock_take` and `lock_steal`\ntake a `lane`, which is *recorded and never consulted*. A static-lane deploy and a\nweb-lane deploy touch the same box, so two sessions holding \"different lanes\" at\nonce is the exact accident the lock exists to prevent.\n\n> **Measured, and it did not work as intended.** This tier was added after Haiku 4.5\n> scored 0.936 on `full` — near-saturation on the cheapest available model. A\n> same-model A/B (3 tasks each) put `lanes` at **0.911** against `full`'s **0.936**:\n> the right direction, and safety did fall (0.955 → 0.911) with liveness rising to\n> 1.00, but a 0.025 gap is smaller than the spread within the tier itself\n> (0.842–0.957). **`fleet-lock` remains an easy environment.** See *Difficulty,\n> honestly* below.\n\n**2. `*` does not cross `/`.** `api/app/*.py` does not cover\n`api/app/routes/orders.py`; `api/app/**` does. `fnmatch` gets this wrong, and the\nfailure is silent: every claim quietly widens to a subtree and the holder never\nlearns they over-reserved.\n\nThe control suite carries both mistakes as *plausible wrong answers* — the\nreference with exactly one rule broken — and asserts they score **below** a\nceiling. Not just \"cheats lose\" but \"near-misses lose too\": if a near-miss scored\nlike the reference, that trap would not be exercised and the trace would not test\nthe rule it claims to.\n\nNote that the two fail on **different axes**. `fnmatch_glob` over-claims, so it is\n*over*-safe (safety 1.00) and blocks legitimate work (liveness 0.89).\n`expired_is_free` is *under*-safe (safety 0.92). One reward, two distinguishable\nfailure modes.\n\n## Three tiers\n\nDifficulty rises by what a claim can *express*, not merely by trace length — so a\nmodel can be trained up a gradient instead of dropped in at full difficulty.\n\n| tier | sessions | patterns | ops | glob trap | lane trap |\n|---|---|---|---|---|---|\n| `basic` | 2 | exact paths only | 28 | — | — |\n| `standard` | 3 | `+ *` globs | 44 | yes | — |\n| `full` | 4 | `+ **`, bare dirs, trailing `/` | 64 | yes | — |\n| `lanes` | 4 | all | 76 | yes | yes |\n\n`basic` has no glob patterns at all, so the glob trap does not exist there and is\nnot planted — the tier tests the lock state machine and trivial overlap alone.\n\nThat the tiers differ in **kind** rather than in name is asserted, not assumed: the\ncontrol suite scores the `fnmatch_glob` near-miss on both ends and requires it to be\nindistinguishable from the reference on `basic` and punished on `full`.\n\n```\ntier gradient (fnmatch_glob): basic 1.00 -> full 0.88\n```\n\nIf those two numbers ever converge, the tiers have stopped meaning anything and the\nsuite fails.\n\n## Difficulty, honestly\n\nHaiku 4.5 — the cheapest model available — scores **~0.91–0.94** here across tiers.\nThe reference is 1.00. That is a narrow band with little training signal in it, and\na stronger model would likely sit higher still.\n\n**Use this environment as an eval or a regression check, not as an RL training\ntarget.** It discriminates a broken registry from a working one very sharply\n(degenerate strategies score 0, the reference scores 1.00), which makes it a good\ncorrectness harness. It does not discriminate a good model from a great one.\n\nThe likely reason is the domain: a state machine with seven documented rules is\nsomething current models are simply good at. Two attempts to harden it — a fourth\ntier and an adversarially-planted trap — moved the number by 0.025. Further tiers\nwould probably be fighting the domain rather than fixing the environment.\n\nA note on method, since it cost a run to learn: the `per_lane_locks` near-miss (the\nreference with only that rule broken) scores 0.85, and that was read as evidence the\ntrap would bite. A real model scored 0.91. **A synthetic mutation measures whether\nbreaking a rule costs points — which is close to tautological — not whether a model\nwill break it.** The near-misses remain valuable as regression guards. They are not\ndifficulty predictions.\n\n## Verification\n\n```bash\npython3 tests/test_grader.py     # no dependencies, any interpreter\npython3 tests/test_taskset.py    # needs verifiers; skips cleanly without it\npython3 tests/test_runtime.py    # needs verifiers + docker; skips cleanly without\n```\n\n`test_taskset.py` additionally asserts that every trace contains the traps its tier\ndeclares — and that a glob-free tier contains *no* glob claim, so the gradient\ncannot leak — that traces differ across tasks, and that **both** factors of the\nreward carry signal in every trace — a trace whose ground truth is all-allow or all-refuse has a dead\nfactor and would silently reward a degenerate strategy.\n\nBoth currently pass, against `verifiers` 0.3.1 on Python 3.12.13.\n\n## Provenance\n\nDistilled from a coordination tool in daily use on a shared working tree, where the\nfailure being prevented is concrete: one agent's commit sweeping up another's\nhalf-finished edit. The rules here are the ones that were learned by getting them\nwrong first.\n\n## Usage\n\n```bash\nuv pip install -e .\nuv run eval fleet_lock -n 3\nuv run eval fleet_lock --env.taskset.tier basic\nuv run eval fleet_lock --env.agent.runtime.type docker   # local, free\n```\n\nNote `eval`, not `vf-eval` - the latter is the legacy evaluator and cannot load a\nv1 taskset. The module id uses an underscore.\n","encoding":"utf-8","truncated":false,"total_bytes":7591},"status":null}