{"data":{"kind":"file","path":"README.md","version_id":"ai1crrs3jaq2tmlll1x2osxe","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":7130,"modified_at":"2026-10-04T16:39:45.665000","content_hash":"f8f8f8ab1d3ab94e4677c13343a416afe1e2ae444b3fe48a16283d672c316eea"},"entries":[],"content":"# pact\n\nA coding agent drives a LIBERO robot arm by **writing Python programs** over deterministic primitives, scored on the simulator's own goal predicate. It never emits joint torques: it writes a program, runs it, reads structured failures, and rewrites. At T0–T2 it never sees an image. At T3 it gets two RGB-D cameras and finds everything itself. CPU-only.\n\nThe primitives ship as an installed package on the agent's `PYTHONPATH`, so any harness that can run a shell command works.\n\n## A rollout\n\nEach rollout gets its own container and simulator daemon. `Task.setup` starts `python -m pact_sim`, which compiles the LIBERO scene (~8 s) and listens on a unix socket. The agent runs `robot-python solution.py` as often as it likes; every run talks to the same daemon, so **the world persists across programs**, checked per episode by hashing `qpos` at each connection boundary.\n\n`Task.finalize` asks the daemon for a verdict and checks it answers for this episode's scene and seed. The dispatch table has no verb that writes state, so `qpos[bowl] = plate_position` is not in the search space.\n\n**Threat model.** The agent is root in the container that also runs the daemon. The cheap routes are closed at every tier:\n- `verdict` answers only the secret the host gave the daemon. It reaches the daemon in a boot file deleted on read, so BDDL and seed never appear on the daemon's command line.\n- `robot-python` is a venv with no simulator in it, and the source tree is gone.\n- At T3, LIBERO's scene files are deleted once the scene has compiled, and egress is framework-only.\n\nEvery verdict is signed with a key the daemon never sends anywhere, so a swapped interpreter cannot forge a positive one. A deliberate agent can still ptrace the daemon. So `tamper_*` metrics flag tool calls that reach for the simulator, and the daemon journals every refused probe. Moving the daemon into its own sandbox is the planned fix.\n\n## Tasks and tiers\n\n28 tasks from LIBERO: all of `libero_goal` and `libero_spatial`, and 8 of `libero_object` (butter and chocolate pudding are too small from the fixed camera for `find()`), in the public dataset [`aryanmadhavverma/pact-tasks`](https://huggingface.co/datasets/aryanmadhavverma/pact-tasks). Scenes live in the image `prime/aryanmadhavverma/pact-sim:m16`. Its build fails unless all 30 scenes boot, unprojection checks out through both cameras, and every taskset seed keeps libero_spatial's spatial relations. Six tasks ship a frozen hand-written solution, which `validate` runs with no model.\n\n| tier   | given                                         | must still do                |\n| ------ | --------------------------------------------- | ---------------------------- |\n| T0     | ground-truth object pose                      | grasp choice → IK → sequence |\n| **T1** | an H×W mask, plus `locate()` for the geometry | the same                     |\n| T2     | the mask, no `locate()`                       | the unprojection too         |\n| T3     | two RGB-D cameras + calibration, finger encoders | recognition and segmentation too |\n\nAt T3 there is no `find` and no segmentation. `look(camera)` returns colour, depth and calibration for `agentview` (fixed) or `wrist` (on the hand), and saves both images under `/workspace/views`. Every grasp check uses finger width alone, never contacts or object names. The frozen T3 reference is classical vision only, and solves `put_the_bowl_on_the_plate` on 15/15 held-out seeds (`validate_t3.toml`).\n\n## Measured\n\n**T3, 28 tasks, held-out seeds 1001-1002, one run (pact 0.4.0, image m14, bash harness, 50 turns).**\n\n| arm | model | success |\n| --- | --- | --- |\n| T3, cameras + image tool* | `claude-opus-5.5` | 86% (48/56) |\n| T3, cameras + image tool* | `gpt-6-astra` | 86% (24/28) |\n| T3, cameras + image tool* | `gpt-6-sol` | 69% (36/52) |\n| T3, cameras (as shipped) | `claude-opus-5.5` | 86% (24/28) |\n| T1, same run | `claude-opus-5.5` | 79% (22/28) |\n| T1, same run | `gpt-6-sol` | 75% (21/28) |\n| T3, blind | `Qwen3.5-9B` | 0% (0/56) |\n| T3, random commands, no model | | 0% (0/56) |\n\n\\*An earlier build also gave the model a `view(path)` tool that showed it images as pictures. It made no difference (Opus 86% either way): with only the arrays, agents segment in code, thresholding colour and depth. It was removed, so every model gets the same interface. Removing perception costs frontier agents nothing on LIBERO's clean renders. A harder T3 would need\ndepth noise, look-alike objects and occlusion.\n\n**Earlier T1 results:**\n\nT1 on the ten `libero_goal` tasks, 30 turns, on images before m12 (whose `locate()` rejects stray depth blobs). Denominators differ because runs were pooled across seed sets.\n\n| model             | harness     | solved |     |\n| ----------------- | ----------- | ------ | --- |\n| `gpt-6-astra`     | claude_code | 32/40  | 80% |\n| `gpt-6-astra`     | bash        | 25/41  | 61% |\n| `grok-4.6`        | bash        | 18/40  | 45% |\n| `claude-opus-4.6` | bash        | 8/20   | 40% |\n| `grok-4.6`        | claude_code | 14/40  | 35% |\n| `claude-sonnet-5` | bash        | 22/67  | 33% |\n| `claude-sonnet-5` | claude_code | 21/64  | 33% |\n| `gpt-5.2`         | bash        | 16/56  | 29% |\n| `deepseek-v4-pro` | claude_code | 9/40   | 22% |\n| `claude-opus-4.6` | claude_code | 3/18   | 17% |\n| `glm-4.7`         | bash        | 6/40   | 15% |\n| `deepseek-v4-pro` | bash        | 6/43   | 14% |\n| `gpt-5.2`         | claude_code | 5/38   | 13% |\n| `glm-4.7`         | claude_code | 5/40   | 12% |\n\n![baseline](baseline.png)\n\nThe noise floor is wide: two identical runs scored 0/12 and 3/12, so at n=40 only gaps of ~32 points are detectable.\n\n## Running\n\nFrom a clone of the repository root:\n\n```bash\nuv sync\nuv pip install -e environments/pact\nuv run validate @ configs/pact/validate.toml   # frozen references, no model\nuv run validate @ configs/pact/validate_t3.toml --taskset.seed-offset 1000   # T3 reference\nuv run eval @ configs/pact/eval.toml           # one episode per task\n```\n\nOr from the Hub, into any venv: `uv pip install pact --extra-index-url https://hub.primeintellect.ai/aryanmadhavverma/pact/install/simple/`.\n\nBoth run on Prime sandboxes. Any OpenAI-compatible endpoint works, e.g. Claude direct: `--client.base-url https://api.anthropic.com/v1/ --client.api-key-var ANTHROPIC_API_KEY -m claude-sonnet-5`.\n\n| `--env.taskset.*` | default | |\n| --- | --- | --- |\n| `tier` | `t1` | `t0`, `t1`, `t2`, `t3` |\n| `depth_noise` | 0 | T3: depth noise std at 1 m (grows with range²) |\n| `prompt_suffix` | \"\" | appended verbatim; red-team and ablation runs only |\n| `suites` / `only` | all | filter by suite / `task_id` |\n| `seeds`, `seed_offset` | 3, 0 | seeds are `seed_offset+1 …`; 1000 is the held-out range |\n| `legible` | true | false strips failure evidence (ablation) |\n| `orientation_hint` | true | false withholds `GRIPPER_DOWN` (ablation) |\n| `image` | the m16 ref | a local tag after `make -C scripts/pact build` |\n| `dataset`, `revision` | pinned | the task rows |\n\nThe image is built from `scripts/pact/` (`make push` builds amd64 on Prime); the rows from `generators/pact/`.\n","encoding":"utf-8","truncated":false,"total_bytes":7130},"status":null}