{"data":{"kind":"file","path":"README.md","version_id":"k7zjw8w1zz08c1uiunlt2dvp","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2480,"modified_at":"2026-10-31T11:34:57.023000","content_hash":"243f91824d6537eb3f321a422bce7b83185dbea3bdcbe49549b93b12d469daa6"},"entries":[],"content":"# Tool Call Repair Benchmark (`tool-call-repair`)\n\n## Overview\nAutonomous agents frequently produce malformed or schema-violating tool/function calls when operating at high velocity or utilizing smaller models. Common failure modes include stringified numbers, missing required parameters, hallucinated arguments, invalid enum variants, and unescaped quotes.\n\n`tool-call-repair` is a deterministic, single-turn reinforcement learning and evaluation benchmark testing an agent's capability to repair corrupted or schema-violating tool calls into strictly valid, executable tool invocations conforming to formal OpenAI/Anthropic tool schemas.\n\n## Benchmark Structure\n- **Tasks**: 50 procedurally generated benchmark instances spanning 5 core agent domains:\n  1. `database_query` / `sql_executor`\n  2. `payment_transfer` / `stripe_checkout`\n  3. `web_scraper` / `browser_navigate`\n  4. `cloud_provisioner` / `container_deploy`\n  5. `calendar_scheduler` / `meeting_invite`\n- **Corruption Modes Evaluated**:\n  - `stringified_numbers`: Passing strings where integers or floats are strictly required.\n  - `missing_required_args`: Omitting parameters defined in the schema's `required` list.\n  - `hallucinated_extra_args`: Adding unpermitted keys when `additionalProperties: false`.\n  - `invalid_enum_value`: Utilizing casing/variant mismatches outside the declared enum.\n  - `stringified_boolean`: Python literals or string booleans (`\"true\"` vs `true`).\n  - `flattened_nested_object`: Omitting container objects for nested parameters.\n  - `malformed_json_arguments`: Unescaped internal quotes and syntax faults.\n  - `wrapped_explanation_markdown`: Conversational text and unformatted markdown fences.\n\n## Scoring Rubric\nThe environment scores outputs deterministically on a scale of `[0.0, 1.0]` across three dense reward functions:\n1. `tool_syntax_reward` (Weight: 0.2): Valid JSON syntax containing `name` (string) and `arguments` (object).\n2. `schema_conformance_reward` (Weight: 0.4): Complete parameter schema compliance (type validation, enum adherence, required fields present, no forbidden properties).\n3. `argument_fidelity_reward` (Weight: 0.4): Exact or structural value match against target ground-truth parameters.\n\n## Usage\n\n```bash\nprime env install crescentic-osmosis/tool-call-repair\n```\n\n```python\nimport verifiers as vf\n\nenv = vf.load_environment(\"crescentic-osmosis/tool-call-repair\")\ndataset = env.get_dataset()\nprint(f\"Loaded {len(dataset)} evaluation tasks\")\n```\n","encoding":"utf-8","truncated":false,"total_bytes":2480},"status":null}