{"data":{"kind":"file","path":"README.md","version_id":"r62616s3xj63xv2ihx2zekak","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":4214,"modified_at":"2026-07-04T16:35:36.483000","content_hash":"52c8c4581ecb107d427751c07587d6cf2216ce95dfd22532ae227f0f50322d5e"},"entries":[],"content":"# harvey-lab-classic\n\n### Overview\n- **Environment ID**: `harvey-lab-classic`\n- **Short description**: Verifiers implementation of [Harvey LAB](https://github.com/harveyai/harvey-labs), an open-source benchmark for evaluating LLM agents' ability to perform legal work in realistic environments.\n- **Tags**: legal, tools, sandbox, multi-turn, train, eval\n\n### Datasets\n- **Primary dataset**: [`irfanjamil/Harvey-LAB`](https://huggingface.co/datasets/irfanjamil/Harvey-LAB)\n- **Training split**: `train`\n- **Evaluation split**: `eval`\n- **Dataset selection**: fixed by the environment and not configurable through environment arguments.\n- **Document representation**: source documents are staged into the sandbox as text-backed files at their original paths under `/workspace/documents`.\n\n### Environment Specifics\n- **Type**: multi-turn tool-use environment for legal tasks.\n- **Model-facing tools**: `bash`, `read`, `write`, `edit`, `glob`, `grep`.\n- **Tool execution**: tool calls are executed in a Prime sandbox, with one sandbox per rollout.\n- **Sandbox image**: fixed by the environment and not configurable through environment arguments.\n- **Output expectations**: exact `.docx`/`.xlsx` filenames directly under `/workspace/output`.\n- **Rubric overview**: reward follows the method used in the original Harvey LAB: expected deliverables are parsed inside the sandbox with `parse-doc`, then the fixed `deepseek-v4-flash` judge model evaluates the agent's outputs against the task's fixed set of criteria.\n\n### Quickstart\nRun an evaluation with default settings:\n\n```bash\nprime eval run harvey-lab-classic\n```\n\nConfigure model and sampling:\n\n```bash\nprime eval run harvey-lab-classic -m openai/gpt-4.1-mini -n 1 -r 1\n```\n\nNotes:\n- Use `-a` / `--env-args` to pass environment-specific configuration as a JSON object.\n- `prime eval run` saves results automatically; do not add upload opt-out flags unless that is intentional.\n\n### Environment Arguments\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `max_turns` | int | `200` | Maximum model turns. |\n| `judge_parallelism` | int | `6` | Concurrent criterion judge calls. |\n\n### Secrets\n\n`PRIME_API_KEY` and `DEEPSEEK_API_KEY` may be exported in the shell, or placed in a `.env` file in the current working directory used to launch Python. Exported environment variables take precedence over `.env` values.\n\n### User Costs\n\nEval costs come from three sources:\n\n- Prime sandbox usage: one rollout-scoped sandbox is created for each rollout. The sandbox uses 1 CPU core, 2 GB RAM, and 5 GB disk, with a fixed 17 minute lifetime limit. Sandbox pricing information is available in the [Prime Intellect sandbox docs](https://docs.primeintellect.ai/sandboxes/overview).\n- Inference for the model being evaluated. Lowering `max_turns` reduces the maximum number of model turns per rollout, which can limit inference cost and usually shortens rollout duration, reducing sandbox lifetime and sandbox cost.\n- Inference for the fixed `deepseek-v4-flash` LLM-as-a-judge reward model. Each rollout is associated with a list of task criteria, and the judge evaluates each criterion individually. A rollout with N criteria therefore uses N judge calls, so judge inference cost scales with rollouts times criteria per rollout.\n\nThe cost per rollout is capped by the model turn limit, the sandbox lifetime limit, and the use of a relatively cheap fixed judge model. The environment code does not add any other model-inference calls.\n\n### Reward\n\nA rollout's reward is the fraction of task criteria judged passing. This scalar is the training signal used for RL training.\n\n### Metrics\n| Metric | Meaning |\n| ------ | ------- |\n| `lab_criteria_passed` | Passing criteria count. |\n| `lab_criteria_total` | Total criteria count. |\n| `lab_criterion_pass_rate` | Same scalar as rollout reward. |\n| `lab_missing_deliverables` | Expected deliverables not found exactly under `/workspace/output`. |\n| `lab_deliverable_errors` | Expected deliverables that failed sandbox parsing. |\n| `lab_judge_calls` | Criterion judge calls made. |\n| `lab_tool_calls` | Total exposed tool calls. |\n| `lab_sandbox_lifetime_seconds` | Approximate rollout sandbox lifetime. |\n","encoding":"utf-8","truncated":false,"total_bytes":4214},"status":null}