{"data":{"kind":"file","path":"README.md","version_id":"la0j3bfbsirt86ynqvtc3sye","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6953,"modified_at":"2026-08-02T17:01:36.365000","content_hash":"8351df54eeabcc03cd774b09aea6a93adb379d144f57238570928f3022adb978"},"entries":[],"content":"# RAGent Deep Search\n\n### Overview\n\n- **Environment ID**: `ragent_deep_search`\n- **Short description**: A Verifiers v1 environment for training and evaluating whether an agent can perform deep search over a defined knowledge base or data source, inspect relevant documents, and answer multi-part factual questions.\n- **Tags**: `verifiers-v1`, `agentic-search`, `retrieval`, `LanceDB`, `rubric-judge`\n\nThe agent receives a question and can iteratively call retrieval tools before returning a natural-language answer. A model-based judge grades that answer against the question's per-example rubric.\n\n### Datasets\n\n- **Primary dataset**: The private [`diegi97/ragent-rubrics`](https://huggingface.co/datasets/diegi97/ragent-rubrics) dataset generated by the repository's [`data_pipelines`](../../data_pipelines/) project.\n- **Default split**: `test`. Set `taskset.split` to select another Hub split.\n- **Local development**: Set `taskset.dataset_path` to a local `.jsonl` file. Local files are detected automatically, `taskset.split` is ignored, and the data source is read from `prepare_config.data_source` in the sibling `metadata.json` file.\n\nEach dataset row has the following shape:\n\n```json\n{\n  \"data_source\": \"gitlab_handbook\",\n  \"entity\": \"CVE\",\n  \"question_type\": \"Deep single-entity question\",\n  \"question\": \"...\",\n  \"rubric\": [\n    {\n      \"criterion\": \"The answer identifies ...\",\n      \"doc_ids\": [45515, 94156]\n    }\n  ],\n  \"doc_ids\": [45515, 94156]\n}\n```\n\n`data_source`, `entity`, `question_type`, `question`, `rubric`, and `doc_ids` are required for Hub records. `data_source` selects the matching LanceDB table. For local pipeline output, the loader adds `data_source` from the sibling `metadata.json`; any row-level value is replaced by this authoritative metadata value. Criterion-level document IDs are metadata for the expected supporting sources.\n\n### Task\n\n- **Type**: Multi-turn, tool-using question answering.\n- **Output**: A final natural-language answer that addresses every requested fact and is grounded in the retrieved documents.\n- **Available tools**:\n  - `search(queries)`: searches for several queries in one call.\n  - `read(doc_ids)`: returns the full text of selected documents.\n  - `text_scan(pattern, ...)`: scans the corpus for a fixed string or regular expression.\n- **Rubric**: Each dataset criterion becomes `criterion_01`, `criterion_02`, and so on. Criteria are graded in concurrent batches. Each criterion receives a binary `no`/`yes` score, and the example reward is their weighted mean.\n\nThe retriever opens the local LanceDB namespace at:\n\n```text\n${LANCEDB_DB_URI:-lancedb}/<namespace>\n```\n\nThe matching index must already exist. The repository's `ragent_core/scripts/sync_lancedb_gcs.py` utility can be used to mirror an existing namespace from GCS when applicable.\n\nFor dense or hybrid retrieval, prefer setting `RAGENT_EMBEDDING_SERVICE_URL`\nand, for reranked hybrid retrieval, `RAGENT_RERANKER_SERVICE_URL`. Concurrent\nrollouts can then share one model deployment rather than loading embedding and\nreranking weights in the environment process. Pure BM25 retrieval does not use\neither service.\n\n### Quickstart\n\nRequirements:\n\n- Python 3.12 or newer and `uv`.\n- Access to the Hugging Face dataset, or a local JSONL file in the schema above.\n- A populated local LanceDB namespace.\n- API credentials for both the rollout model and rubric judge.\n\nInstall the environment dependencies:\n\n```bash\ncd environments/ragent_deep_search\nuv sync\n```\n\nStore credentials and the local database location in an uncommitted `.env` file:\n\n```dotenv\n# Prime Inference\nPRIME_API_KEY=your-prime-api-key\n\n# Direct OpenAI, if used instead\nOPENAI_API_KEY=your-openai-api-key\n\nHF_TOKEN=your-hugging-face-token\nLANCEDB_DB_URI=/absolute/path/to/lancedb\n```\n\nUpdate [`evaluation.toml`](evaluation.toml), especially `model`, `taskset.split`, `taskset.tools.namespace`, and the judge settings. Set `taskset.dataset_path` only when overriding the default Hub dataset or using a local JSONL file. Validate the resolved Verifiers v1 configuration without making model calls:\n\n```bash\nuv run --env-file .env eval @ evaluation.toml --dry-run\n```\n\nThen run the evaluation:\n\n```bash\nuv run --env-file .env eval @ evaluation.toml\n```\n\nThe run writes its resolved configuration and `traces.jsonl` under:\n\n```text\noutputs/ragent_deep_search--<model>--<harness>/<run-id>/\n```\n\n### Environment Arguments\n\nTaskset, tool, and judge arguments belong under `[taskset]`, `[taskset.tools]`, and `[taskset.task.judge]` respectively.\n\n| Argument | Type | Default | Description |\n| --- | --- | --- | --- |\n| `taskset.dataset_path` | `str \\| Path` | `\"diegi97/ragent-rubrics\"` | Hugging Face dataset ID or local JSONL file containing questions and rubrics. |\n| `taskset.split` | `str` | `\"test\"` | Hub split to load. Ignored when `dataset_path` is a local JSONL file. |\n| `taskset.num_tasks` | `int` | `100` | Maximum number of records loaded by the taskset. The top-level eval `num_tasks` can select a smaller run. |\n| `taskset.tools.namespace` | `str` | `\"default\"` | LanceDB namespace used by the retrieval tools. |\n| `taskset.tools.device` | `str \\| null` | `null` | Optional compute device used while loading the retriever. |\n| `taskset.tools.retrieval_mode` | `str` | `\"bm25\"` | Retrieval pipeline: `bm25`, `dense`, `hybrid`, or `hybrid_reranked`. |\n| `taskset.task.judge.model` | `str` | Verifiers judge default | Model used to grade rubric criteria. |\n| `taskset.task.judge.base_url` | `str` | Prime Inference URL | Judge API endpoint. Configure it independently from `[client]`. |\n| `taskset.task.judge.api_key_var` | `str` | `\"PRIME_API_KEY\"` | Name of the environment variable containing the judge API key. |\n| `taskset.task.judge.max_criteria` | `int \\| null` | `4` | Maximum criteria per judge call. `null` grades all criteria in one call. Must be at least `1` when set. |\n| `taskset.task.judge.view` | `str` | `\"last_reply\"` | Portion of the agent trace shown to the judge. |\n\nCommon top-level evaluation controls are:\n\n| Argument | Description |\n| --- | --- |\n| `model` | Rollout model evaluated by the environment. |\n| `num_tasks` | Number of loaded tasks to evaluate. |\n| `num_rollouts` | Independent rollouts per task. |\n| `max_concurrent` | Maximum concurrent rollouts. |\n| `max_turns` | Maximum agent/tool turns per rollout. |\n| `sampling.max_tokens` | Maximum generated tokens per model response. |\n| `push` | Whether to upload the completed evaluation. |\n\nDo not use a top-level `tools` or `--tools` option. Tool configuration is nested under `[taskset.tools]` in Verifiers v1.\n\n### Metrics\n\n| Metric | Range | Description |\n| --- | --- | --- |\n| `rubric` | `[0, 1]` | Weighted mean of all criterion scores for the example. |\n| `rubric/criterion_NN` | `{0, 1}` | Binary score emitted for an individual criterion. |\n\nBecause this task currently has one reward function, the trace's aggregate reward is the same as the `rubric` metric.\n","encoding":"utf-8","truncated":false,"total_bytes":6953},"status":null}