{"data":{"kind":"file","path":"README.md","version_id":"kacwdlhx4nxkq5l9eirpuitk","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2732,"modified_at":"2026-08-17T22:22:07.482000","content_hash":"03b2f72e0afbb47400f22290b62464ae1a16a4f5fcf8bb84bce85ec610d04f94"},"entries":[],"content":"# supabase-evals\n\nVerifiers OpenCode environment that clones a **pinned** [supabase/evals](https://github.com/supabase/evals.git) commit into the sandbox, moves the current rollout out of that clone, boots **platform-lite + MCP** in-container, and scores with the same **`EVAL.ts`** modules as `pnpm eval`.\n\n## Pin\n\n| Field | Value |\n|---|---|\n| Repo | `https://github.com/supabase/evals.git` |\n| Commit | `0f6097ac8b0966b9b6df61f1e7572f02fb454660` (`0f6097a Refine copy (#152)`) |\n\nOverride per row via `info.git_sha`.\n\n## MCP wiring (`info.mcp_mode`)\n\nMatches the original harness (not “MCP on every task”):\n\n| `mcp_mode` | When | OpenCode gets |\n|---|---|---|\n| `full` | tools-mode eval, or local-stack with `hostedProject: true` | MCP → platform-lite (account/database/…) |\n| `docs` | local-stack / `interface: cli` without hosted project | MCP docs-only (`search_docs`) — no `apply_migration` |\n\nRebuild flags from `PROMPT.md` with `python scripts/build_dataset.py`.\n\n## Layout inside the sandbox\n\n```\n/sandbox-workspace/task/          # agent cwd — this rollout only\n/opt/supabase-evals/.vf-score/<id>/  # PROMPT + local/remote seed (inside clone for module resolve)\n/opt/supabase-evals/              # pinned clone for pnpm + platform-lite (evals/ wiped)\n```\n\n`EVAL.ts` is copied to `.vf-score/<id>/` so the sidecar can `import()` it at boot, then **deleted** (along with `.git` and clone README/CONTRIBUTING) before OpenCode starts. Scoring uses the in-memory scorer — the file is not copied back. Worker logs:\n\n```\nscorer hide eval=<id> eval_ts=absent git=absent sidecar=ok\nscorer hide-check eval=<id> eval_ts=absent git=absent sidecar=ok\n```\n\n`eval_ts=present` on hide-check is a WARNING (hide failed or the agent restored the file); scoring still runs.\n\n## Setup\n\n```bash\ncd environments/supabase_evals\nuv run python scripts/build_dataset.py   # from monorepo evals/ → dataset.jsonl\nuv pip install -e .\n```\n\n## Eval\n\n```bash\nexport AK=sk-local\nuv run vf-eval supabase-evals \\\n  -k AK -b http://127.0.0.1:8080/v1 -m Ornith-1.0-35B \\\n  --api-client-type openai_chat_completions \\\n  -n 3 -r 1 -c 1 -s --disable-tui \\\n  -a '{\"sandbox_type\":\"openshell\"}'\n```\n\n## Scoring\n\n`supabase_evals_lib/sidecar/supabase-eval-sidecar.ts` is uploaded into the sandbox and run against `/opt/supabase-evals` after `pnpm install` (framework filter **and** root workspace so scorer deps like `common-tags` resolve). Score fixtures live under `/opt/supabase-evals/.vf-score/<id>/` so Node module resolution walks into the clone’s `node_modules`. `EVAL.ts` is imported at sidecar start and removed from disk before the agent runs; `/score` does not re-read it. OpenCode MCP points at in-container platform-lite on `127.0.0.1`.\n","encoding":"utf-8","truncated":false,"total_bytes":2732},"status":null}