{"data":{"kind":"file","path":"README.md","version_id":"o95rj48w2otyl0kl83s2titj","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5845,"modified_at":"2026-07-04T16:35:41.302000","content_hash":"64bdf5f2bfbef91651090740a7c15d1c9860647a188a82024965df76685c1304"},"entries":[],"content":"# harvey-lab-rlm\n\n### Overview\n- **Environment ID**: `harvey-lab-rlm`\n- **Short description**: Verifiers implementation of [Harvey LAB](https://github.com/harveyai/harvey-labs), an open-source benchmark for evaluating LLM agents' ability to perform legal work in realistic environments, using a Recursive Language Model runtime.\n- **RLM runtime**: the root model receives the task instructions and one model-facing tool, `call_python_repl`. That tool drives a persistent Python worker inside a Prime sandbox. The worker exposes LAB-specific helpers and can delegate focused, stateless sub-LLM requests through `llm_batch`.\n- **Tags**: legal, rlm, sandbox, multi-turn, train, eval\n\n### Datasets\n- **Primary dataset**: [`irfanjamil/Harvey-LAB`](https://huggingface.co/datasets/irfanjamil/Harvey-LAB)\n- **Training split**: `train`\n- **Evaluation split**: `eval`\n- **Dataset selection**: fixed by the environment and not configurable through environment arguments.\n- **Document representation**: source documents are staged into the Python namespace as a read-only `documents` dictionary mapping document name to document text, not as user-editable workspace files.\n\n### Environment Specifics\n- **Type**: multi-turn RLM environment for legal tasks.\n- **Root model-facing tool**: `call_python_repl`.\n- **Python namespace**: `instructions`, `documents`, `skills`, `expected_deliverables`, sandbox-local `read`, `write`, `bash`, host-routed `llm_batch`, and the mutable `answer` completion signal.\n- **Sub-LLMs**: stateless, tool-free calls made only when root-model code invokes `llm_batch`. They receive the task objective plus the context the root model explicitly includes.\n- **Tool execution**: root Python code and sandbox-local helpers execute in a Prime sandbox, with one sandbox per rollout.\n- **Sandbox image**: fixed by the environment and not configurable through environment arguments.\n- **Output expectations**: exact `.docx`/`.xlsx` filenames directly under `/workspace/output`.\n- **Rubric overview**: reward follows the method used in the original Harvey LAB: expected deliverables are parsed inside the sandbox with `parse-doc`, then the fixed `deepseek-v4-flash` judge model evaluates the agent's outputs against the task's fixed set of criteria.\n\n### Quickstart\nRun an evaluation with default settings:\n\n```bash\nprime eval run harvey-lab-rlm\n```\n\nConfigure model and sampling:\n\n```bash\nprime eval run harvey-lab-rlm -m openai/gpt-4.1-mini -n 1 -r 1\n```\n\nNotes:\n- Use `-a` / `--env-args` to pass environment-specific configuration as a JSON object.\n- `prime eval run` saves results automatically; do not add upload opt-out flags unless that is intentional.\n\n### Environment Arguments\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `max_turns` | int | `200` | Maximum root-model turns. |\n| `sub_model` | str \\| null | `null` | Model used by `llm_batch`; when unset, sub-LLM calls reuse the root model. |\n| `judge_parallelism` | int | `6` | Concurrent criterion judge calls. |\n\n### Secrets\n\n`PRIME_API_KEY` and `DEEPSEEK_API_KEY` may be exported in the shell, or placed in a `.env` file in the current working directory used to launch Python. Exported environment variables take precedence over `.env` values.\n\n### Sandbox Image\n\nThe environment owns the sandbox image and always uses `irfanjamil10/harvey-lab-rlm-sandbox:0.1.0`. The image contains Pandoc, LibreOffice Writer/Calc, document fonts, `parse-doc`, and the Python dependencies required by the bundled DOCX/XLSX skills.\n\n### User Costs\n\nEval costs come from four sources:\n\n- Prime sandbox usage: one rollout-scoped sandbox is created for each rollout. The RLM sandbox uses 1 CPU core, 2 GB RAM, and 5 GB disk, with a fixed 17 minute sandbox lifetime limit and a 20 minute outer rollout timeout. Sandbox pricing information is available in the [Prime Intellect sandbox docs](https://docs.primeintellect.ai/sandboxes/overview).\n- Inference for the root model being evaluated. Lowering `max_turns` reduces the maximum number of root-model turns per rollout, which can limit inference cost and usually shortens rollout duration, reducing sandbox lifetime and sandbox cost.\n- Inference for sub-LLM delegation through `llm_batch`. These calls happen only when the root model invokes `llm_batch` from the Python worker. If `sub_model` is unset, the root model is reused; if `sub_model` is set, sub-LLM cost follows that model's pricing and scales with the number of delegated requests.\n- Inference for the fixed `deepseek-v4-flash` LLM-as-a-judge reward model. Each rollout is associated with a list of task criteria, and the judge evaluates each criterion individually. A rollout with N criteria therefore uses N judge calls, so judge inference cost scales with rollouts times criteria per rollout.\n\nThe cost per rollout is capped by the root-model turn limit, the sandbox lifetime limit, the outer rollout timeout, and the use of a relatively cheap fixed judge model. The environment code does not add any other model-inference calls.\n\n### Reward\n\nA rollout's reward is the fraction of task criteria judged passing. This scalar is the training signal used for RL training.\n\n### Metrics\n| Metric | Meaning |\n| ------ | ------- |\n| `lab_criteria_passed` | Passing criteria count. |\n| `lab_criteria_total` | Total criteria count. |\n| `lab_criterion_pass_rate` | Same scalar as rollout reward. |\n| `lab_all_pass` | Whether all non-skipped criteria passed. |\n| `lab_missing_deliverables` | Expected deliverables not found exactly under `/workspace/output`. |\n| `lab_deliverable_errors` | Expected deliverables that failed sandbox parsing. |\n| `lab_judge_calls` | Criterion judge calls made. |\n| `lab_rollout_duration_seconds` | Approximate model-generation rollout duration. |\n| `lab_sandbox_lifetime_seconds` | Approximate rollout sandbox lifetime. |\n","encoding":"utf-8","truncated":false,"total_bytes":5845},"status":null}