{"data":{"kind":"file","path":"README.md","version_id":"ns2jdsi76tz9j8jl2e63odq6","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":7723,"modified_at":"2026-07-08T15:42:27.437000","content_hash":"f30f954d1532bdd01f564e081f92557c8508de23b157df77e6e2f05e5edca4ca"},"entries":[],"content":"# cua-world\n\n[![arXiv](https://img.shields.io/badge/arXiv-2604.06126-red.svg?style=for-the-badge)](https://arxiv.org/abs/2604.06126)\n[![Docs](https://img.shields.io/badge/Docs-Read-blue?style=for-the-badge&logo=readthedocs&logoColor=white)](https://cmu-l3.github.io/gym-anything/docs/)\n[![Interactive Paper](https://img.shields.io/badge/Interactive-Paper-purple?style=for-the-badge)](https://cmu-l3.github.io/gym-anything/interactive_paper.html)\n[![GitHub](https://img.shields.io/badge/GitHub-Code-black?style=for-the-badge&logo=github)](https://github.com/cmu-l3/gym-anything)\n\nCUA-World is the benchmark built with\n[Gym-Anything](https://github.com/cmu-l3/gym-anything)\n([paper](https://arxiv.org/abs/2604.06126)): real desktop, web, and mobile\napplications running in full VMs, with the applications selected by their\neconomic (GDP) impact. Each rollout boots a real Linux, Windows, or Android\nguest in the cloud. The policy sees screenshots and acts with mouse and\nkeyboard, and a checklist-based VLM verifier with task-specific privileged\ninformation scores the trajectory.\n\n## Overview\n\n- **Environment ID**: `cua-world`\n- **Scale**: 12,866 tasks across 246 environments (200+ applications, covering all 22 major occupation groups)\n- **Platforms**: Linux (206 environments), Windows (29), Android (11)\n- **Splits**: `train` (10,397), `test` (2,469), `long_horizon` (201, one long-horizon task per application, often hundreds of steps: CUA-World-Long in the paper)\n- **Agent**: a reference agent scaffold (`Qwen3VLAgent`) runs inside the environment; the framework samples your chosen model through it\n- **Scoring**: each task's VLM checklist with integrity checks (the paper's verifier); the task's programmatic `verifier.py` optional\n- **Runtime**: one VM per rollout (QEMU or Android emulator) inside a Modal sandbox; nothing runs on your machine\n\nExample applications: GIMP, Apache OpenOffice, Ardour, Rocket.Chat, Apache\nOFBiz (Linux), Garmin BaseCamp, Epi Info (Windows), QField, Sygic GPS\n(Android).\n\n## Install\n\n```bash\nprime env install pranjal2041/cua-world\n```\n\n## Run\n\nThe default configuration runs one GIMP task end to end:\n\n```bash\nprime eval run pranjal2041/cua-world -n 1 -r 1\n```\n\nPick the policy with `-m` as usual. Select tasks, applications, or a split\nwith `-a`:\n\n```bash\nprime eval run pranjal2041/cua-world -m google/gemini-3-flash-preview -n 10 -r 1 \\\n  -a '{\"env_names\": \"all\", \"split\": \"long_horizon\", \"max_turns\": 15}'\n```\n\nFor prime-rl training, reference the environment by id and forward the same\narguments:\n\n```toml\n[[orchestrator.train.env]]\nid = \"cua-world\"\nargs = { env_names = [\"gimp_env_all_fast\"], split = \"train\", max_turns = 15 }\n```\n\nBecause the agent rebuilds a windowed prompt each turn, use\n`trajectory_strategy = \"branching\"` when training.\n\n### Credentials\n\nTwo secrets are required (on the hub they live in this environment's Secrets\ntab, for local runs export them):\n\n- `GEMINI_API_KEY`: the grader. Every episode is judged by a VLM checklist\n  (gemini-3.5-flash by default). Without it, rollouts still run and bill VM\n  time but score 0 with an error recorded.\n- `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET`: the VMs run in Modal sandboxes\n  under this account (locally, `modal token set ...` works too).\n\nRunning locally with `vf-eval` additionally needs a key for the policy\nendpoint, passed with `-b`/`-k`:\n\n```bash\nvf-eval cua-world -m gemini-3.5-flash \\\n  -b https://generativelanguage.googleapis.com/v1beta/openai/ -k GEMINI_API_KEY \\\n  -n 1 -r 1 -a '{\"task_ids\": [\"add_border\"]}'\n```\n\n## Environment arguments\n\nThe ones most runs touch:\n\n| Argument | Default | Description |\n| --- | --- | --- |\n| `env_names` | `\"gimp_env_all_fast\"` | Application environment name(s), or `\"all\"` |\n| `split` | `\"all\"` | `train`, `test`, `long_horizon`, or `all` |\n| `task_ids` | `None` | Optional task allowlist within the selected environments |\n| `max_turns` | `15` | Model-turn budget per rollout |\n| `agent` | `\"Qwen3VLAgent\"` | Reference agent scaffold (e.g. `GeminiQwen3Agent` for the gemini-tuned variant) |\n| `agent_args` | `{\"model\": \"gemini-3.5-flash\", \"temperature\": 1.0}` | The agent's own arguments |\n\nAdvanced (defaults are right for almost everyone):\n\n| Argument | Default | Description |\n| --- | --- | --- |\n| `verifier_mode` | `\"vlm_checklist\"` | Set `None` to use each task's programmatic `verifier.py` instead |\n| `vlm_model` / `vlm_base_url` / `vlm_api_key_var` | gemini-3.5-flash on Google's endpoint | Change the grader model |\n| `runner` | `\"modal\"` | `qemu_native` / `docker` / `avd_native` run VMs locally instead |\n| `remote_url` | `None` | Run rollouts on a gym-anything remote cluster |\n| `use_cache` / `cache_level` / `use_savevm` | `True` / `\"post_start\"` / `False` | VM checkpoint behavior |\n| `surface` | `\"raw\"` | `\"raw\"` or `\"verified\"` task surface |\n| `seed` / `max_examples` | `0` / `None` | Reset seed, dataset row cap |\n\n## Scoring and metrics\n\nEach task ships a `vlm_checklist.json`: weighted completion items plus\nintegrity checks, written with privileged information from the task's own\nsetup (ground truth the agent never sees). The judge scores every item with\npass, partial, or fail. Failing any integrity item zeroes the score,\nfollowing the paper.\n\n| Metric | Weight | Meaning |\n| --- | --- | --- |\n| `task_reward` | 1.0 | Final reward (0 to 1) |\n| `verifier_score` | 0.0 | Final checklist score (0-100) after the integrity gate |\n| `verifier_completion` | 0.0 | Checklist score (0-100) before the integrity gate |\n| `verifier_integrity` | 0.0 | 1.0 if every integrity check passed |\n| `verifier_passed` | 0.0 | 1.0 on a perfect, integrity-clean checklist |\n| `actions_executed` / `parse_errors` | 0.0 | Actions executed in the VM, malformed tool calls |\n\nEvery saved sample carries the full verdict in `info.verifier` (per-item\nverdicts with the judge's evidence, sub-scores, overall reasoning) and the\ncomplete multi-turn trajectory (every screenshot and action). Scoring runs\nwhile the VM is still alive. A rollout whose grading fails is recorded with\nreward 0 and the error in `info.verifier.error` instead of aborting.\n\n## Cost\n\nOne rollout is one VM (about 5 CPU / 12 GB, billed on your Modal account).\nConcurrency (`-c`) bounds how many VMs run at once. The first rollout in a\nnew environment pays a one-time provisioning boot (minutes to tens of\nminutes) and saves a checkpoint to a shared Modal volume, after which\nrollouts start in a few minutes. A few environments need a one-time asset\nfetch (`<env>/scripts/fetch_data.sh`, or `MANUAL_DOWNLOAD.md` for manual\nassets) before their tasks pass.\n\n## Reference results\n\nFrom the paper (test split, Gemini 3 Flash as judge):\n\n| Model | Avg. score | Pass rate |\n| --- | ---: | ---: |\n| Gemini 3 Flash | 50.1 | 22.6% |\n| Kimi-K 2.5 | 37.1 | 12.8% |\n| Qwen3-VL-2B | 12.7 | 1.6% |\n\nOn `long_horizon` the strongest model reaches 7.5% pass rate at a 500-step\nbudget, and 27.5% (GPT-5.4) at 2,000 steps.\n\n## How it works\n\nThis package is a thin declaration over the gym-anything library. Each\nrollout runs the reference agent's `step()` verbatim, and the framework\nsamples the model through the agent's `llm_call` seam, so prompts, history\nhandling, and parsing are identical to a local run. Disk images and\ncheckpoints persist in a Modal volume across rollouts. See the\n[hubs documentation](https://cmu-l3.github.io/gym-anything/docs/extras/hubs/)\nfor the full design.\n\n## Local development\n\nInside the gym-anything repo, install gym-anything editable first, then this\nshell without deps (so it resolves your checkout instead of the pinned tag):\n\n```bash\nuv pip install -e \".[modal,prime-rl,benchmark]\"      # from repo root\nuv pip install -e extras/hubs/prime/cua_world --no-deps\n```\n","encoding":"utf-8","truncated":false,"total_bytes":7723},"status":null}