{"data":{"kind":"file","path":"README.md","version_id":"pptyzsdkznlx0j45r9mu8w4z","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":8548,"modified_at":"2026-10-27T14:38:42.853000","content_hash":"acefc6ed9934c06ab2e8091a96f8fbbe1a20931cf51bf7a2243e65e4eb15a59e"},"entries":[],"content":"# Biomni GeneTrail v1 — evidence-grounded biomedical agent RL\n\nRelease: **v2.0.1** (package version `2.0.1`; native Verifiers v1 API).\n\n**An original Prime Intellect / Verifiers v1 taskset inspired by the *style of biomedical multi-tool reasoning* in Stanford's Biomni project.** It is **not** Biomni, not Biomni-R0 weights, not the Biomni-E1 tool environment, not a reproduction of Biomni-Eval1, and not a clinically valid benchmark.\n\nThis is a **synthetic, deterministic, offline agent environment** for multi-step interpretation of fictional Mendelian genetics data. No HF downloads, API keys, network access, clinical guidance, patient information or wet-lab protocols are needed to generate cases or compute rewards.\n\n## What makes it an agent task (not a single-turn QA wrapper)?\n\nEach episode gives an opaque synthetic case ID. An agent must use an MCP toolset to bring together phenotypes, candidate variant lists, gene annotation, population frequency, sequencing QC, transmission and allele phase. It then submits a machine-checkable final decision **and tool-grounded evidence IDs**.\n\nThe agent must choose among six genes and roughly 13 synthetic alleles. Hard negatives include overlapping phenotype clusters, very low-quality apparent variants, common alleles, inherited AD candidates, and *cis* instead of *trans* AR pairs. Three different inheritance cases are generated: AD, AR compound heterozygous, and X-linked recessive (synthetic male).\n\n### Five MCP tools\n\n| Tool | Evidence surfaced | Why it is necessary |\n|---|---|---|\n| `case_overview(case_id)` | Presentation, synthetic phenotype IDs, sex | Establish case context |\n| `list_candidates(case_id)` | Candidate variant IDs, genes, genotypes | Find entities to inspect |\n| `gene_profile(gene_id)` | Gene/phenotype signature and inheritance model | Distinguish nearby phenocopies |\n| `variant_assessment(case_id, variant_id)` | AF, allele effect, read depth, MAPQ, balance | Reject common and failed-QC calls |\n| `segregation_report(case_id, gene_id)` | Family transmission and phase | Verify de novo, trans and hemizygous cases |\n\nAll tool data is generated in process from the symbolic world. A tool cannot return `gold_gene`, `gold_variants` or private role labels. No LLM is needed to grade a rollout.\n\n### Reward\n\n| Component | Weight | Rule |\n|---|---:|---|\n| Gene match | 40% | Exact synthetic gene ID |\n| Causal variants | 30% | Jaccard score against exact variant-ID set (1 for perfect set) |\n| Inheritance | 15% | Exact `AD`, `AR`, or `XLR` |\n| Source provenance | 15% | Required evidence IDs are both cited in JSON **and observed in TOOL messages** in the v1 trace |\n\nRejects malformed JSON, duplicate variants, duplicate evidence IDs, extra fields and unwanted trailing prose. Citations typed by the assistant without a tool observation do not earn provenance credit. When trace/tool events cannot be recovered, evidence score fails closed to zero but the remaining 85% still reflects scientific correctness.\n\nExample final schema (illustrative only — NOT a true answer):\n\n```json\n{\"gene\":\"SIMG004\",\"variants\":[\"SV-001-82744\",\"SV-003-18471\"],\"inheritance\":\"AR\",\"evidence_ids\":[\"EV-0123456789ABCD\"]}\n```\n\n## Install and test\n\nRequirements: Python 3.11+, uv, Prime CLI; optional model provider credentials for a real rollout. Version target: **Verifiers 0.4.x** with native v1 tasksets. Use an MCP-capable harness. The built-in `null` harness is useful for this task because it supports an MCP-only agent and does not expose an unnecessary shell.\n\n```bash\n# Unzip, cd to the directory containing pyproject.toml\nuv sync --extra test\nuv run pytest -q\n\n# Confirm CLI command flags with your installed versions\nuv run vf-eval --help\nuv run vf-validate --help\n\n# Model-free validation (does not require a model API key)\nuv run vf-validate biomni-gene-trail-v1 -n 8 --runtime.type subprocess\n\n# Real tool-using evaluation. Supply a model supported by your provider.\n# Set its corresponding API key via the CLI's documented method.\nuv run vf-eval biomni-gene-trail-v1 \\\n  -m openai/gpt-4.1-mini -n 10 \\\n  --env.agent.harness.id null \\\n  --env.agent.runtime.type subprocess \\\n  --env.taskset.split eval \\\n  --env.taskset.num-tasks 100\n```\n\n**Important:** Verifiers 0.4 defaults to a Prime remote runtime unless explicitly overridden. `--env.agent.runtime.type subprocess` selects local execution. For fair evaluation, keep the agent on the MCP-only `null` harness, not a coding harness with Python access to the local generator.\n\nCLI flags can change. Check `vf-eval --help` and the [v1 taskset documentation](https://github.com/PrimeIntellect-ai/verifiers/blob/main/docs/v1/tasksets.md) if your installed version differs.\n\n### Push to Environment Hub\n\n```bash\nuv tool install prime\nprime login\n# From the environment package directory, after testing your chosen visibility:\nprime env push -p . --visibility PUBLIC --plain\n```\n\nThe distribution includes a `biomni_gene_trail_v1` discovery entrypoint alongside the implementation package. Public visibility releases the submitted source code.\n\nPushing to the Hub **does not guarantee payment**. Match your work to an open/unclaimed bounty; read the task's exact acceptance criteria, apply for application-only bounties if needed, and get the designated review/claim process before relying on payment.\n\n## Controls and splits\n\nDefault `num_tasks=240`, `split=train`, `start=0`. Supported split values are `train`, `eval`, `stress`. Each has 10,000 deterministic case IDs; generation is on demand and local. Example alternate slice:\n\n```bash\nuv run vf-eval biomni-gene-trail-v1 -n 20 \\\n  --env.agent.harness.id null \\\n  --env.agent.runtime.type subprocess \\\n  --env.taskset.split stress \\\n  --env.taskset.start 500 \\\n  --env.taskset.num-tasks 100\n```\n\n`start` and `num_tasks` control a contiguous deterministic slice. The splitting salt changes observations/variant IDs, but the *gene catalog and task-generating rules are shared*. Therefore these splits are **held-out cases, not held-out biological mechanisms**. Strong generalization claims require new distinct phenomena, out-of-distribution templates and external expert validation.\n\n## Quality / reproducibility\n\n`pytest` covers thousands of programmatically generated scenarios, all three inheritance cases, no gold labels in tool outputs, deterministic generation, malicious response formats, fabricated assistant-only evidence, and direct v1 task method tests. `scripts/release_audit.py` produces a JSON report for a repeatable generator audit.\n\n```bash\nuv run python scripts/release_audit.py --n 1000 --output quality-report.json\n```\n\nUnit tests can run with a small **test-only Verifiers shim** if the SDK is unavailable, to validate our own data and scoring. **This does not establish compatibility with the real Prime evaluation service**. Run `vf-validate` and a genuine model rollout with the actual SDK before submission. No such end-to-end remote run is claimed here.\n\n## Security and limitations\n\n- Synthetic phenotypes (`SYNHP:`), genes (`SIMG`) and variant IDs (`SV-`) are fictional; do not use for diagnosis, treatment or wet-lab decisions.\n- Not clinically validated; no real-world sensitivity, specificity or calibration results.\n- This is an intentionally **transparent public RL task**. Source code contains deterministic generator and grading rules. A shell-access agent that can inspect the installed package may reverse-engineer labels. Evaluate with **MCP-only** tools; high-stakes blind benchmarking requires inaccessible server-side truth and an independent audit.\n- Literal evidence IDs can be reconstructed from public code by an adversarial agent. The reward checks the actual rollout's tool messages, but environment security also depends on an untampered trace from the runtime.\n- No model benchmark scores are asserted without actual model evals. A high oracle unit-test score demonstrates only correct implementation, not difficulty for frontier models.\n\n## Literature and attribution\n\n- Biomni project: https://github.com/snap-stanford/Biomni\n- Biomni-R0 technical report link supplied in the task context: https://biomni.stanford.edu/blog/biomni-r0-technical-report/\n- Verifiers native v1 authoring: https://github.com/PrimeIntellect-ai/verifiers/blob/main/docs/v1/tasksets.md\n- Prime CLI: https://github.com/PrimeIntellect-ai/prime\n- Prime Intellect environment program: https://www.primeintellect.ai/blog/scaling-environments-program\n\nMIT licensed, original synthetic implementation; no proprietary clinical dataset redistributed.\n","encoding":"utf-8","truncated":false,"total_bytes":8548},"status":null}