{"data":{"kind":"file","path":"README.md","version_id":"wjcl4wawzzhh7ferf4lyqwsk","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3039,"modified_at":"2026-09-20T16:48:06.007000","content_hash":"af38802df76220f4843257bb20509190df0e1c8319e0f4e3e3c90f7df1885c0b"},"entries":[],"content":"# AutomationBench-Verified\n\nA benchmark for evaluating AI agents on realistic business workflows.\n\n- **White Paper:** https://arxiv.org/abs/2604.18934\n- **GitHub:** https://github.com/zapier/AutomationBench\n- **Artificial Analysis:** https://artificialanalysis.ai/evaluations/automationbench-aa\n\nLearn more at [zapier.com/benchmarks](https://zapier.com/benchmarks) or run it on the [Prime Intellect Environments Hub](https://app.primeintellect.ai/dashboard/environments/zapier/AutomationBench).\n\n## Changes compared to AutomationBench\n\n- The tools are exposed as native tools to the models with their full schema, not as stringified JSON or prose\n  - This heavily relies on top-level `allOf`, `anyOf` and `oneOf`. Some open model tool parsers do not parse those values correctly. [vLLM PR](https://github.com/vllm-project/vllm/pull/53729), [SGLang PR](https://github.com/sgl-project/sglang/pull/36626)\n- Fixes some of the APIs to be called; some of the loaded tools for some of the tasks as well as the graders.\n- Builds on top of verifiers v1, removes all the legacy (v0) code.\n\n## Experimental LLM judge\n\nSet `taskset.task.judge` to a native Verifiers `JudgeConfig` to use LLM-only grading.\nThe built-in `RubricJudge` receives the original instructions, initial and final\napplication state, and transcript. Existing keyword assertions do not determine its reward.\nThe endpoint and credentials use the native Verifiers judge configuration and Prime CLI defaults.\n\nThe pilot also reads this configuration from TOML. For example, `examples/judges/glm.toml` contains:\n\n```toml\n[taskset.task.judge]\nmodel = \"internal/glm-5.3-fast\"\n```\n\nThe judge uses the bundled completion rubric and the built-in plain-JSON parser. The rubric and\noutput format are fixed. Change `model` to select another judge; `examples/judges/luna.toml`\ncontains a Luna configuration. Run three constructed cases with:\n\n```sh\nuv run --no-sync python scripts/judge_pilot.py /tmp/automationbench-glm-pilot \\\n  --config examples/judges/glm.toml \\\n  --case hr_correct --case sla_equivalent_wording --case sla_wrong_account_links\n```\n\nPass `--fixtures-dir PATH` to reuse the saved states and instructions from an earlier pilot.\n\nThe bundled rubric has one `full_completion` criterion, giving **binary** reward: 1 only when\nall material requirements are satisfied, otherwise 0. The existing `partial_credit` field carries\nthis result; it is not comparable to the original fractional assertion score. The native judge\nrecords the criterion score in `trace.metrics[\"rubric/full_completion\"]` and its explanation,\nrequest, and response in `trace.info[\"judge_calls\"]`, with usage in `trace.extra_usage`.\nAPI and parsing errors propagate as errors rather than incorrect-work scores.\n\nThe bundled rubric resolves from the installed package, independent of the working directory.\nLeaving `judge` unset preserves the original assertion grader. The pilot uses one shared completion\ncriterion, not per-task checklists, and does not establish judge reliability across the full benchmark.\n","encoding":"utf-8","truncated":false,"total_bytes":3039},"status":null}