{"data":{"kind":"file","path":"README.md","version_id":"xrakqdi5mkkvcd5kt39vskzx","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3075,"modified_at":"2026-09-17T15:23:54.608000","content_hash":"203d27a1bcfc15bfa29c29e18badabb17d9fc514274acaeb150f064d282e2ab8"},"entries":[],"content":"# ALE Linux CLI\n\nA native Verifiers v1 taskset for the 105-task Linux CLI subset of\n[Agents Last Exam](https://agents-last-exam.org/leaderboard). Of the 105 pinned\ntasks, 9 with known upstream defects are excluded by default, leaving 96 tasks\nin the default selection. Nested tasks are included except the ones with defects.\n\n## Setup\n\nFrom the `residency-environments` repository root:\n\n```bash\ncd environments/ale_cli\n```\n\n`uv run` creates and synchronizes the environment on the first run.\n\nThe evaluation environment pins Verifiers to `0.3.2.dev95`, the latest published\ndevelopment build as of September 15, 2026. Prerelease dependencies are enabled\nfor the Verifiers development build and its dependencies. From the\nrepository root, synchronize the pinned environment with\n`uv sync --project environments/ale_cli --all-extras --locked`.\n\nHidden references live in the gated Hugging Face dataset\n[`agents-last-exam/agents-last-exam-reference`](https://huggingface.co/datasets/agents-last-exam/agents-last-exam-reference);\nrequest access, then authenticate locally:\n\n```bash\nhf auth login\nprime login\n```\n\nFor noninteractive runs, set the credentials in the evaluator process:\n\n```bash\nexport HF_TOKEN=hf_...\n# Required only for business_finance/pe_screening_memo_1:\nexport OPENAI_API_KEY=sk_...\n```\n\nHosted runs need `HF_TOKEN` as a Prime secret linked to the environment.\n`OPENAI_API_KEY` is required only for `business_finance/pe_screening_memo_1`,\nwhose pinned host-side scorer uses `gpt-4o-mini`. These credentials never enter\ntask data or evaluation traces.\n\n## Run\n\n```bash\nuv run eval \\\n  ale-cli \\\n  --env.agent.harness.id codex \\\n  --env.agent.runtime.type prime \\\n  --env.agent.runtime.vm true \\\n  --model deepseek/deepseek-v4-flash \\\n  --num-rollouts 1 \\\n  --max-concurrent 16 \\\n  --env.interception.multiplex 16\n```\n\nThis runs one rollout for every task in the default selection. To run one task,\nadd `--env.taskset.task-id <task-id>`.\n\nPrime VM mode is required because some ALE tasks enforce restricted network\npolicies, and Prime applies per-task egress allow/block lists only to VM\nsandboxes.\n\n### Taskset configuration\n\n| Field | Type | Default | Description |\n|---|---|---|---|\n| `task_id` | `str \\| None` | `None` | Select one canonical ALE Linux task ID; `None` applies the configured skip filters. |\n| `image` | `str \\| None` | `None` | Override the default runtime image for all selected tasks. |\n| `skip_nested_tasks` | `bool` | `false` | Skip the six tasks that require nested-container or VM-backed execution. |\n| `skip_bad_tasks` | `bool` | `true` | Skip the nine tasks with known upstream task, reference, scoring, or runtime-artifact defects. |\n| `timeout_multiplier` | `float` | `1.0` | Multiply each task's upstream agent timeout; tasks without a timeout remain uncapped. |\n| `resource_multiplier` | `float` | `1.0` | Multiply CPU, memory, and disk after task-specific minimums. |\n| `task.max_stage_concurrent` | `int` | `16` | Maximum hidden-reference files staged concurrently during scoring to avoid Hugging Face resolver rate limits. |\n","encoding":"utf-8","truncated":false,"total_bytes":3075},"status":null}