{"data":{"kind":"file","path":"README.md","version_id":"txllwh339dmbvebg4n3jjmyk","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":9481,"modified_at":"2026-10-22T14:37:58.618000","content_hash":"c4b00f4da2e42227d4378f69949e82d71946b8a0969ece8ed9ca14b7085c98c9"},"entries":[],"content":"# ChemResearch Lab v1\n\n![ChemResearch Lab cover](assets/lab_cover.png)\n\n**Open-source scientific-reasoning and experiment-design environment for Prime Intellect / Verifiers v1.**\n\n[![License: Apache-2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)\n![Python](https://img.shields.io/badge/Python-3.11%2B-blue)\n![Verifiers](https://img.shields.io/badge/Verifiers-v1-informational)\n\n> Status: **community-built candidate / local validation passed**. Not an accepted or assigned Prime Intellect bounty. Official evaluation and bounty review have **not** been performed. Research data are **synthetic**, not from a physical laboratory.\n\n## Motivation\n\nA model completing these tasks must infer latent chemical quantities from imperfect observations, satisfy physical relations, select an informative follow-up measurement, and return machine-graded scientific results. Unlike a science-fact multiple-choice benchmark, each case contains numerically distinct experimental measurements generated from a model with controlled noise.\n\nThe environment provides:\n\n- Four chemistry task families, including inverse problems and experimental-design decisions.\n- Procedural datasets: 240 tasks by default; configurable seeds, size and train/eval/challenge splits.\n- **Deterministic numerical reward** in `[0, 1]`, with separate accuracy, experiment-design, and consistency components.\n- No judge model, external API, API key, web service, or physical laboratory control.\n- A reference (oracle) solver to audit reward math, plus independent observation-only baselines that do **not** access the oracle's latent parameters.\n- Unit tests, demo scripts, a model-evaluation config, and documented limitations.\n\n## Benchmark tasks\n\n| Family | Inference goal | Experiment-design component | Physical basis |\n|---|---|---|---|\n| UV–Vis mixture unmixing | Concentrations of two synthetic analytes from noisy spectral readings | Additional wavelength that maximizes increase in `det(XᵀX)` | Beer–Lambert linear absorbance, fixed 1 cm path |\n| First-order kinetics | Rate constant `k` and half-life from time-series absorbance | Sampling time with maximum `|dA/dk|` under fixed-amplitude / homoscedastic assumption | `A(t) = baseline + A0 exp(-kt)` |\n| Arrhenius inference | Activation energy and extrapolated rate/half-life | Follow-up temperature maximizing reciprocal-temperature leverage | `ln(k) = ln(A) - Ea/(RT)` |\n| Ideal buffer design | Stock aliquots, dilution water and predicted pH | Satisfy pH, material balance and final volume simultaneously | Henderson–Hasselbalch; ideal analytical concentrations |\n\nThe prompt includes all calibrations and observable data. The numeric target values are not included in prompt text. The actual chemistry is deliberately idealized: see **Scope & limitations**.\n\n## Quick start\n\nPrerequisites: Python 3.11–3.14, [`uv`](https://docs.astral.sh/uv/), and a Prime Intellect model endpoint/account only if you want an actual model evaluation.\n\n```bash\n# From this directory\nuv venv\nuv pip install -e .\n\n# Verify the scientific simulator and scoring offline (no model needed)\nuv pip install pytest\nuv run pytest -q\nuv run python examples/demo_offline.py\nuv run python examples/run_baselines.py\n```\n\nTo evaluate a language model through **Verifiers v1**:\n\n```bash\n# The environment package should be installed in the active uv environment\nuv run vf-eval chemresearch-lab-v1 -n 16 \\\n  --env.taskset.split eval \\\n  --env.taskset.seed 2026 \\\n  --env.agent.harness.id null \\\n  --model YOUR_MODEL_ID\n```\n\nAlternatively, replace `YOUR_MODEL_ID` in `eval.toml` and run:\n\n```bash\nuv run vf-eval @ eval.toml --dry-run\nuv run vf-eval @ eval.toml\n```\n\n**Important:** the commands and adapter follow the official v1 documentation, but the full Verifiers CLI run has **not** been executed in the provided build environment because fetching Verifiers from PyPI was blocked by network/DNS restrictions. The offline scientific code and tests **have** run successfully. This distinction is intentional, so that we do not present an unverified integration as verified.\n\n## Upload to Prime Intellect's Environments Hub\n\nAfter confirming your Prime Intellect account, username, and successful `vf-eval` run:\n\n```bash\nprime login\n# Inside this directory:\nprime env push\n```\n\nThe published environment is uploaded to **your** Prime Intellect account. Confirm the current Hub upload instructions in the [official documentation](https://docs.primeintellect.ai/tutorials-environments/create).\n\n## JSON contract and reward\n\nModels must return **one JSON object**, no surrounding explanation. Each family has its own schema:\n\n**UV–Vis**\n\n```json\n{\"concentration_a_uM\": 11.5, \"concentration_b_uM\": 7.2, \"confirm_wavelength_nm\": 575, \"predicted_absorbance\": 0.314}\n```\n\n**Kinetics**\n\n```json\n{\"rate_constant_per_min\": 0.024, \"half_life_min\": 28.88, \"next_measurement_min\": 50, \"predicted_absorbance\": 0.24}\n```\n\n**Arrhenius**\n\n```json\n{\"activation_energy_kj_mol\": 55.0, \"predicted_rate_constant_s_inv\": 0.022, \"predicted_half_life_s\": 31.51, \"next_temperature_K\": 343.15}\n```\n\n**Buffer**\n\n```json\n{\"acid_stock_mL\": 37, \"base_stock_mL\": 21, \"water_mL\": 142, \"predicted_pH\": 4.65}\n```\n\nThese examples show only the schema; they are **not** answers to a specific generated task.\n\nThe grader uses smooth Gaussian-shaped relative/absolute error rewards for scientific estimates, with consistency and design-credit bonuses. Invalid JSON, undefined/unsafe numeric values, and missing or impossible quantities receive zero. Rewards are truncated to `[0,1]`.\n\n| Family | Inference / scientific objective | Design | Internal numerical consistency |\n|---|---:|---:|---:|\n| Spectral unmixing | 70% | 15% | 15% (gated by accuracy) |\n| Kinetics | 55% | 20% | 25% (gated) |\n| Arrhenius | 72% | 15% | 13% (gated) |\n| Buffer | 72% | 14% volume balance | 14% (gated) |\n\n**Anti-gaming:** a model cannot earn full points solely by producing internally self-consistent but chemically wrong values. See `grade_case()` in `core.py` for the auditable exact implementation. There is no subjective LLM grader or hidden reward endpoint.\n\n## Dataset splits and reproducibility\n\nIn `ChemResearchConfig`:\n\n- `split`: `train` / `eval` / `challenge` (independent pseudo-random substreams).\n- `seed`: base integer seed; same seed + family + split regenerates identical prompt/ground truth.\n- `num_tasks`: requested finite dataset size (1 to 100,000), spread evenly across task families.\n- `challenge` uses higher simulated measurement noise in spectral/kinetic/Arrhenius tasks.\n\nTo prevent benchmark contamination in research, use fixed public evaluation seeds and an additional held-out private seed suite. The package does not promise that procedural code being public prevents evaluation-set leakage or memorization.\n\n## Quality checks\n\n```bash\npython -m compileall -q chemresearch_lab_v1\npytest -q\npython examples/run_baselines.py\n```\n\nThe tests cover: oracle sanity checks over 300 generated cases, seed/split separation, JSON strictness, physically consistent buffer mass balance, spectroscopy basis independence, half-life identities, degradation after wrong answers, finite score ranges under extreme outputs, and the ability of a measurement-only baseline to solve held-out challenge cases.\n\nFor honest model benchmarking, publish performance from at least two distinct model sizes on an evaluation split, inspect individual traces and error distributions, and compare to the included numerical baseline. Do **not** claim a model score based only on the oracle/baseline scripts.\n\n## Scope & limitations\n\n- Entirely simulated scientific tasks, without real measurements, lab instruments or hazardous-synthesis instructions.\n- Only ideal Beer–Lambert, first-order kinetics, Arrhenius and Henderson–Hasselbalch physics; e.g. real buffers may require activity coefficients and temperature-dependent pKa.\n- Kinetics inference uses an approximately correct log-linear model with noise in absorbance; this baseline can become biased at low absorbances.\n- A standalone, primarily single-turn v1 taskset. It does not yet include tool-mediated multi-turn laboratory exploration, wet-lab safety validation, or frontier model benchmarking.\n- The animated lab cover is AI-generated conceptual artwork, **not** photographic evidence of experiments.\n\n## Structure\n\n```text\nchemresearch_lab_v1/\n├── chemresearch_lab_v1/\n│   ├── __init__.py           # exports ChemResearchTaskset\n│   ├── taskset.py            # Verifiers v1 task and reward adapter\n│   ├── core.py               # deterministic physics / data / grader\n│   └── baselines.py          # observation-only reference solver\n├── tests/test_science.py\n├── examples/demo_offline.py\n├── examples/run_baselines.py\n├── assets/lab_cover.png\n├── eval.toml\n├── BOUNTY_SUBMISSION.md\n├── LICENSE\n└── pyproject.toml\n```\n\n## Official references\n\n- [Prime Intellect Verifiers v1 – Tasksets](https://docs.primeintellect.ai/verifiers/v1/tasksets)\n- [Verifiers v1 – Evaluations](https://docs.primeintellect.ai/verifiers/v1/evaluation)\n- [Upload an Environment](https://docs.primeintellect.ai/tutorials-environments/create)\n- [Prime Intellect – Open-Source Environments Program](https://www.primeintellect.ai/blog/scaling-environments-program)\n\n## License\n\nApache-2.0. Attribution is required under the terms of the license. The cover artwork is synthetic.\n","encoding":"utf-8","truncated":false,"total_bytes":9481},"status":null}