{"data":{"kind":"file","path":"README.md","version_id":"tvw7mdfjvwvnnc5en3o1qnbp","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6101,"modified_at":"2026-07-08T05:04:40.989000","content_hash":"32bd323e22dfcf8f2ba4377174c44741f5a8acd9a5867fb740cee5c68a553021"},"entries":[],"content":"# are-gaia2\n\nMeta's [Agents Research Environments](https://github.com/facebookresearch/meta-agents-research-environments) (ARE) / [Gaia2](https://huggingface.co/datasets/meta-agents-research-environments/gaia2) scenarios as a verifiers environment, for RL training and evaluation of tool-using agents.\n\n**Wrap, don't rebuild:** ARE's own apps, event scheduler, and judges run inside the env (ARE is a pip dependency). The adapter replaces exactly one thing — ARE's real-time clock — with a **token-metered simulated clock** so rewards are deterministic and safe for async RL.\n\n### Overview\n- **Environment ID**: `are-gaia2`\n- **Short description**: Multi-app agentic scenarios (email, calendar, messaging, files, …) with scheduled events, notifications, and temporal deadlines, judged against oracle action traces.\n- **Tags**: tool-use, multi-turn, agent, temporal, gaia2, train, eval\n\n### The token-metered clock\n\nARE natively runs a real-time event loop (1 real second = 1 tick) and never pauses while the model thinks, so model latency — and therefore serving hardware and infrastructure load — leaks into scores. That is unusable for async RL: reward would depend on inference-server queue depth. Instead, this adapter advances simulated time per agent turn:\n\n```\ndt = completion_tokens / decode_rate_tps\n   + prompt_tokens / prefill_rate_tps\n   + tool_call_seconds × n_tool_calls\n```\n\nOne **global** rate for every model (never per-model/per-hardware): \"thinking time\" is a pure function of tokens generated. Events fire, notifications arrive, and deadlines expire in ARE's own machinery as the clock advances. A `wait_for_notification` pseudo-tool lets the agent idle until the next scheduled event.\n\n**Consequence:** scores are NOT comparable to the published Gaia2 leaderboard (different clock semantics). Internal comparisons within this harness are valid, and a 5-scenario parity check against ARE's native harness (same model, same scripted judge) agreed 5/5 — see `PORTING.md`.\n\n### Datasets\n- **Primary**: `meta-agents-research-environments/gaia2` (HF) — 1,120 scenarios across capability configs: `execution`, `search`, `adaptability`, `ambiguity`, `time`, `agent2agent`. ~2.5 MB JSON per scenario (fully serialized app universes).\n- **Bundled** (`source=\"bundled\"`): ARE's demo scenarios; used for adapter self-tests, no network needed.\n- **Filesystem data**: scenarios lazily read files from `meta-agents-research-environments/gaia2_filesystem` (HF, ~9 MB). Snapshot it locally and set `DEMO_FS_PATH` — see Setup.\n\n### Task\n- **Type**: multi-turn tool use (55–110 tools per scenario, auto-derived from ARE app APIs)\n- **Rubric**: ARE's own scenario judge. `judge_mode=\"scripted\"` (default) = deterministic `GraphPerEventJudge` matching agent events against oracle events — no LLM, right choice for RL. `judge_mode=\"llm\"` adds soft LLM comparisons (needs a judge endpoint). Do **not** bypass `preprocess_scenario`: without a wired judge, `scenario.validate()` is vacuous.\n- **Reward**: 1.0 if the scenario judge reports success, else 0.0. (Judge rationales expose per-tool agent-vs-oracle counts — dense partial credit is a planned extension; see `PORTING.md`.)\n\n### Setup\n\n```bash\n# 1. ARE is a REQUIRED runtime dep but intentionally undeclared (its\n#    pydantic==2.10.6 pin conflicts with verifiers' floor on paper only):\nuv pip install meta-agents-research-environments\n\n# 2. One-time data warm (quota-friendly: HF anon/free limit is 1000 req/5min):\npython -c \"from datasets import load_dataset; load_dataset('meta-agents-research-environments/gaia2', name='execution')\"\npython -c \"from huggingface_hub import snapshot_download; print(snapshot_download('meta-agents-research-environments/gaia2_filesystem', repo_type='dataset'))\"\n\n# 3. Kill all runtime HF traffic (required for parallel rollouts):\nexport DEMO_FS_PATH=\"<snapshot-path-from-step-2>/demo_filesystem\"\nexport HF_HUB_OFFLINE=1 HF_DATASETS_OFFLINE=1\n```\n\n### Quickstart\n\n```bash\nprime eval run are-gaia2 -m openai/gpt-4.1-mini -n 5 -r 1 \\\n  -a '{\"source\": \"gaia2\", \"capability\": \"execution\", \"judge_mode\": \"scripted\"}'\n```\n\nReference point: gpt-4.1-mini scores ~0.2 on the first 5 `execution` validation scenarios (~170k input tokens per rollout).\n\n### Environment Arguments\n\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `source` | str | `\"gaia2\"` | `\"gaia2\"` (HF) or `\"bundled\"` (ARE demos) |\n| `capability` | str | `\"execution\"` | Gaia2 capability config |\n| `split` | str | `\"validation\"` | HF split |\n| `judge_mode` | str | `\"scripted\"` | `\"scripted\"` (deterministic, no LLM) or `\"llm\"` |\n| `max_scenario_duration` | int | `None` | Override scenario sim-time budget (seconds) |\n| `num_examples` | int | `-1` | Limit scenarios (-1 = all) |\n| `decode_rate_tps` | float | `50.0` | Global nominal decode rate — identical for every model |\n| `prefill_rate_tps` | float | `5000.0` | Global nominal prefill rate |\n| `tool_call_seconds` | float | `2.0` | Fixed simulated cost per tool call |\n| `time_scale` | float | `1.0` | Multiplies each turn's dt (curriculum dial; >1 = tighter deadlines) |\n| `max_turns` | int | `20` | Hard cap on agent turns per rollout |\n| `reward_mode` | str | `\"binary\"` | `\"binary\"`: judge success = 1.0 else 0.0. `\"dense\"`: failures get tool-overlap partial credit in [0, 0.8] (training shaping — keep evals binary) |\n| `scenario_split` | str | `None` | Frozen split, e.g. `\"execution_v1/train\"` (128) / `\"execution_v1/heldout\"` (32). Whole-universe stratified; train on train, report on heldout only |\n\n### Metrics\n\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` | 1.0 if ARE's scenario judge reports success, else 0.0 |\n\n### Status & known gaps\n\nPhase-0 verified: bundled + real Gaia2 scenarios run end-to-end, anti-vacuous judge check passed, 5/5 native-harness score parity. Known gaps (load-time oracle pass blocks the event loop; `agent2agent` scenarios unexamined; multi-turn triggers verified structurally only; Gaia2 data license unverified — keep private): full list and porting runbook in [`PORTING.md`](PORTING.md).\n","encoding":"utf-8","truncated":false,"total_bytes":6101},"status":null}