{"data":{"kind":"file","path":"README.md","version_id":"delio17jjqb54of912uhhahq","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5220,"modified_at":"2026-09-12T00:27:34.661000","content_hash":"233f2a33a5a84502196c5fdd2c8e5e09f08fdaf2b74d42af5830844244d8177e"},"entries":[],"content":"# Long-Horizon Medical Agent Benchmark — Veterinary Clinic Ops\n\n> **Published by [Praesidium Compliance Systems Corporation](https://praesidiumsystems.ai)**\n> · Built on **[Stackhouse](https://github.com/ArjavDesa912/stackhouse)** ([stackhousedb.com](https://stackhousedb.com))\n> · **No LLM-as-judge anywhere** — all 98 tasks are graded by deterministic,\n> fail-closed Python verifiers asserting exact database state, never by a\n> model scoring the transcript.\n\nA 98-task veterinary-clinic operations benchmark running against a **live\nStackhouse BaaS** — packaged as one fully self-contained Docker container\n(Postgres 15 + Stackhouse BaaS + prebuilt Vite frontend + OpenEnv env server).\nNo external services, no volumes, no API keys.\n\n- **Contract**: OpenEnv gym (`/reset`, `/step`, `/state` on port 8000)\n- **Image**: built from `proj/server/Dockerfile` (the unified bundle Dockerfile)\n- **Action space**: one Stackhouse REST call per step\n  (`{method, endpoint, payload, as_user}`)\n- **Reward**: 1.0 iff the task's standalone fail-closed verifier exits 0\n- **Episode isolation**: `reset()` restores a pristine template database\n  (frozen at image build time) in ~1-2s and injects a fresh per-episode nonce\n  (`EP-XXXXXXXX`) plus a +/-3-day-jittered episode date, so memorized answers\n  go stale every episode\n- **Task discovery**: `GET /tasks` (or the TaskProvider routes) lists all 98\n  tasks; select one via `reset(task_id=...)`\n- **v2**: every task is now hard difficulty (v1 was 20 easy / 48 medium / 32\n  hard), seed volume grew ~7.5x, and every aggregate is verified two\n  independent ways (Python filter + SQL) before being compared to the agent's\n  report. See the public sample repo's `BUILD_NOTES.md`/`QC_REPORT.md` for the\n  full v2 changelog and QC transcripts.\n\n## Using the environment (verifiers v1)\n\nThis is a **verifiers v1** package: it exports a `Taskset` (one task per\n`task_id`, 98 total) and a custom `Env` whose `run()` boots a fresh Prime\nSandbox from the pre-registered image and plays the OpenEnv gym contract as\nthe rollout's user. Install and eval:\n\n```bash\nprime env install praesidiumsystems/long-horizon-medical-agent-benchmark-vet-clinic-ops\n\n# 5 tasks x 3 rollouts, default `null` chat harness on the `player` seat:\nuv run eval long-horizon-medical-agent-benchmark-vet-clinic-ops \\\n  -n 5 -r 3 --model google/gemini-3.7-flash\n\n# pick tasks / turn cap / sandbox size via the typed config:\nuv run eval long-horizon-medical-agent-benchmark-vet-clinic-ops \\\n  --env.taskset.task_ids '[\"021_overdue_invoice_flagging\"]' \\\n  --env.player.max_turns 40 \\\n  --env.taskset.sandbox_memory_gb 8\n```\n\nPer-step rewards are summed onto the trace as `openenv_reward` (the final\nverify step carries the 0/1 task score). Requires Prime auth\n(`prime login` / `PRIME_API_KEY`) — the sandbox is created at rollout time\nfrom the image pinned in `proj/build_manifest.json`, no local Docker needed.\n\nDirect OpenEnv access (no verifiers) still works against the same image:\n\n```python\nimport asyncio\nfrom openenv import EnvClient\n\nasync def main():\n    client = await EnvClient.from_docker_image(\n        \"docker.io/arjavdesai912/veterinary_clinic_system:latest\"\n    )\n    async with client:\n        result = await client.reset(task_id=\"021_overdue_invoice_flagging\")\n        print(result.observation.instruction)  # the mission text\n        print(result.observation.nonce)        # EP-XXXXXXXX episode batch code\n        # agent works by emitting Stackhouse REST calls:\n        result = await client.step({\n            \"method\": \"GET\",\n            \"endpoint\": \"/v1/query/veterinary_clinic_system_billing_invoices?limit=5\",\n            \"payload\": None,\n            \"as_user\": \"verifier\",\n        })\n        print(result.reward, result.observation.reward_reason)\n\nasyncio.run(main())\n```\n\nPorts: `8000` env server (OpenEnv API), `8080` Stackhouse BaaS REST,\n`5110` vet-clinic frontend.\n\n## Credentials (baked into the image, intentionally not secret)\n\n- `verifier` login: `rl-admin@rl.local` / `RLVerifier2025!` (read/write API user)\n- `app_admin` login: `admin@pawsclinic.com` / `Demo123!`\n\n## Anti-reward-hacking design\n\n- Every task fails on pristine state (no-op/random baselines proven per task)\n- Fresh nonce + randomized per-episode parameters defeat hard-coded answers\n- Canary sha256 checks prove out-of-blast-radius collections are untouched\n- Verifiers are read-only, fail-closed, and re-hash task files at grade time\n- Grading code lives under `/app/env/grading`, read-only to non-root users;\n  the agent's only surface is the HTTP API\n- Postgres local-socket auth is `peer` and TCP requires scram-sha-256, so a\n  shell user inside the container cannot bypass the BaaS\n\n## How the image is built\n\n`proj/server/Dockerfile` is a unified single-container build: it compiles the\nStackhouse BaaS from source, builds the frontend, seeds Postgres, freezes the\npristine template DB, and installs the env server — no base image prebuild, no\nexternal services, no volumes. You don't need to build it yourself: this\npackage runs as a Prime Sandbox from the image already registered in\n`proj/.build.json`, so `EnvClient.from_docker_image(...)` (above) is all you\nneed — no local Docker required.\n","encoding":"utf-8","truncated":false,"total_bytes":5220},"status":null}