{"data":{"kind":"file","path":"README.md","version_id":"lykr0nhhdwbmwm9m8fmif284","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2411,"modified_at":"2026-09-15T10:53:13.360000","content_hash":"f6c6209b63af48d5e37ebb84957b3149994b2d863a7148fdf1678f920b3b4f97"},"entries":[],"content":"# AI Agent Task Suite (v0.1.0)\n\nA comprehensive benchmark and reinforcement learning (RL) evaluation environment for AI Agent capabilities, built on the Prime Intellect `verifiers` framework.\n\n## Overview\n\nThe `ai-agent-task-suite` assesses autonomous language model agents on critical agentic competencies:\n- **Multi-Step Tool Chaining & Orchestration**: Planning and coordinating sequential API interactions.\n- **Error Recovery & Self-Healing**: Diagnosing transient failures (e.g. database locks, rate limits) and executing backoff/fallback logic.\n- **Safety Guardrails & Destructive Action Interception**: Intercepting high-risk operations (e.g. production resource deletion) and enforcing read-only dry-runs and human approval workflows.\n- **Cross-Source Information Synthesis**: Resolving conflicting constraints across multiple context providers and schedules.\n- **Autonomous Tool & Code Synthesis**: Dynamic code generation with input validation, unit tests, and structured outputs.\n\n## Tasks Included\n\n1. `agent-001-tool-chaining`: Multi-Step API & Data Retrieval Pipeline (Customer operations refund flow).\n2. `agent-002-error-recovery`: Database Transient Error & Exponential Backoff Fallback.\n3. `agent-003-safety-guardrails`: Privilege Guardrail & Destructive Action Interception in Production.\n4. `agent-004-information-synthesis`: Cross-Tool Search and Conflict Resolution (Cross-timezone executive scheduling).\n5. `agent-005-code-generation-eval`: Dynamic Python Tool Synthesizer for statistical metrics.\n\n## Rubric Metrics\n\nThe environment employs a multi-signal rubric:\n1. `metric_tool_chaining_and_plan` (weight: 0.35): Verifies correct tool sequencing, step decomposition, and parameter domain coverage.\n2. `metric_structured_output_schema` (weight: 0.25): Evaluates structural formatting (valid JSON payloads, executable Python AST blocks).\n3. `metric_safety_and_constraint_adherence` (weight: 0.20): Enforces human-in-the-loop approvals, blast radius calculation, and boundary guards.\n4. `metric_reasoning_completeness` (weight: 0.20): Analyzes rationales, trade-off explanations, and failure recovery protocols.\n\n## Usage\n\n### Local Testing\n\n```bash\nuv pip install -e .\npython3 -c \"from ai_agent_task_suite import load_environment; env = load_environment(); print(env)\"\n```\n\n### Prime Evaluation\n\n```bash\nprime eval run philips/ai-agent-task-suite -m meta-llama/Llama-3.2-3B-Instruct\n```\n","encoding":"utf-8","truncated":false,"total_bytes":2411},"status":null}