{"data":{"kind":"file","path":"README.md","version_id":"cnw7hxjb85g4codpk7rnftz4","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":44585,"modified_at":"2026-11-07T09:04:33.850000","content_hash":"56fcb1404e2fe7d44de7524436353b54a590b0ef4d37c4ccc4e70e38ce5fd830"},"entries":[],"content":"# SolarQuoteBench\n\nSolarQuoteBench: an AI benchmark and RL environment for designing, pricing and\nfinancially evaluating solar PV and battery installations. Renamed from solar-eval\non 2026-11-02.\n\nA verifiable-reward RL environment for residential solar and battery quoting. An\nagent is given a site, a system spec, a tariff, or a finished vendor proposal, and\nmust return a numeric answer or a structured audit. Every answer is graded by a\ndeterministic Python oracle, never an LLM judge. The full spec and design\nrationale is in `TASK-V1-ENVIRONMENT.md`; this file is the front door for anyone\ndeciding whether to run or buy it.\n\nThe environment is Python, targeting `verifiers`/`prime-rl`-style harnesses (this\nmarket is Python-only; see `TASK-V1-ENVIRONMENT.md`'s non-negotiables). The\ngrading engine is ported from a real production solar-quoting system,\n[SolarSight](../solarsight), whose own TypeScript test suite is the oracle used to\nverify the port. SolarSight's code is not a runtime dependency of anything here.\n\n## What it measures\n\nEight task families, `src/solarquotebench/tasks/t1_yield_estimate` through\n`t8_full_quote` (T6 to T8 were added in 0.4.0, described after T5). Each has its own generator, grader and at least twelve\nhand-checked instances (`generator.py`'s `curated_instances()`).\n\n- **T1 yield-estimate** (solved control, not a discriminating family): given\n  a site, roof planes and a panel spec, return annual kWh. The value is in\n  three convention traps: compass bearing versus PVGIS's own south-zero\n  aspect convention (a sign error that produces a plausible-looking wrong\n  number, not a crash), hemisphere-dependent optimal tilt, and multiplicative\n  (not additive) composition of system losses. State this plainly up front:\n  against `gpt-5.6-luna` with reasoning on, T1 scores 0.999, 16 of 16 solved,\n  which is ceiling and does not discriminate between models. The traps are\n  real, not fake difficulty: the same model with reasoning turned off falls\n  into the aspect-conversion trap and scores 0.55. So T1 measures whether a\n  reasoning model avoids these traps, which frontier models with reasoning\n  enabled already do; treat it as a control and a regression check, not as\n  evidence of difficulty. Full numbers are in \"What the baseline shows\" below.\n- **T2 orientation-allocate**: allocate a fixed panel budget across multiple\n  roof planes. Every site ships as a matched pair, one instance maximising raw\n  annual yield, the other maximising the dollar value of self-consumption under\n  a tariff and load profile. The two objectives genuinely disagree on some\n  sites (a west-facing plane can win on value while losing on yield); an agent\n  that treats them as the same question fails.\n- **T3 incentive-eligibility**: given a jurisdiction, system spec and install\n  date, return which government incentive schemes apply and their combined\n  value. This is the family with the highest expected failure rate, because an\n  LLM's knowledge of incentive schedules is stale by construction: AU's STC\n  deeming schedule and battery rebate, UK MCS/SEG, IE SEAI grant tiers, US ITC,\n  NZ and CA schemes, including cases where the correct answer is \"not\n  eligible\" and cases straddling a rate step-down.\n- **T4 tou-payback**: given a time-of-use tariff, load profile, system and\n  battery spec, and export limit, return simple payback year and 25-year NPV,\n  graded as two independent dimensions. Tests multi-step arithmetic under price\n  escalation, degradation, export caps and self-consumption timing.\n- **T5 proposal-audit**: given a complete, professional-looking vendor\n  proposal, some clean and some carrying planted errors of a known type and\n  magnitude, find, classify and quantify each error. This is the most novel\n  family: it measures deference to an authoritative-looking wrong number, not\n  raw computation, and clean instances are just as important as dirty ones\n  because a model that flags everything indiscriminately must score badly on\n  them.\n\n### V3 families: the customer's whole quote, starting with current energy use\n\nAdded in 0.4.0 (spec: `TASK-V3-FULL-SCOPE.md`). They share one extra tool,\n`usage_profile`, offered only to these families.\n\n- **T6 consumption-estimate**: the agent reads electricity bills and returns annual\n  consumption in kWh, plus a split by meter or tariff period when asked. Plain\n  arithmetic oracle: the generator prints bills with a forward model and the grader\n  inverts them. Variants: flat rate with the stated tax and daily supply charge to\n  strip; block tiers; per-period TOU kWh; a controlled-load second meter; existing\n  rooftop solar where consumption is import plus generation minus export, not import.\n  Half the instances give only a part-year bill that must be scaled with the seasonal\n  index from `usage_profile`, not by days or months. Tax rates are printed on each bill.\n  Provenance: bill layouts and rates are synthetic (magnitude-anchored to the cached flat\n  tariffs); the seasonal index is a synthetic profile from the environment's own load\n  model (a port of SolarSight's, cross-validated in `tests/test_crossvalidation.py`) and\n  the tool says so. Household-profile estimation from size, pool or EV is deliberately not\n  included: SolarSight's own seed table is a heuristic and no published dataset has been\n  sourced for it. Limits: real households do not follow the synthetic index, so T6 grades\n  whether the agent used the supplied index correctly, not whether it is right for a real\n  home.\n\n- **T7 system-sizing**: choose a panel count per roof plane (in steps, up to a maximum)\n  and a battery size from an offered list, including none, to maximise NPV over the\n  horizon, or to hit a self-sufficiency target at the lowest net cost. Ground truth is the\n  exhaustive optimum over the whole discrete space, valued by the shared quote engine\n  (`domain/quote_engine.py`, a port of the shape of SolarSight's `computeEstimate`).\n  Naive answers (fill the roof, always add a battery, size to 100% of consumption) are\n  stored with their values; every curated instance has at least two that lose 5% or more.\n  Grader: objective score (1% dead band, zero at 10% gap), feasibility, battery decision.\n  The agent has no dispatch simulator, so this grades sizing judgement under the tools it\n  has. Limits: prices and TOU windows are synthetic; the hourly PV shape is newly authored\n  and diverges 2 to 11% from PVGIS hourly data on value-type quantities; the optimum is the\n  optimum of this model.\n\n- **T8 full-quote** (the headline V3 family): from a raw intake (location, electricity\n  bills as in T6, roof planes with a chosen panel count, a proposed battery, a price list,\n  a tariff and projection assumptions) the agent produces the SolarSight-shaped quote:\n  consumption, kWp, annual and monthly production, year-1 energy flows, gross and net cost\n  ranges, incentive line items, annual savings, discounted payback, NPV, IRR and\n  self-sufficiency, for the no-battery and with-battery scenarios. Turn budget 40. A second\n  extended tool, `energy_flows`, returns the year-1 hourly balance for the production and\n  consumption figures the agent passes in, so downstream numbers depend on the agent's own\n  upstream ones. Each field is scored against truth AND against what it should have been given\n  the agent's own upstream numbers (the propagation-aware dimensions, `*_consistency`); the\n  grader names the first step where the agent's own execution broke\n  (`diagnosed_error_mode = \"origin:<step>\"`) and reports the share of wrong steps that were\n  inherited. Ground truth is the shared engine (ported, oracle-verified pieces plus the\n  newly authored hourly PV shape, which has no oracle and diverges 2 to 11% from PVGIS hourly\n  data; the monthly production split is that shape's output, since the cached fixtures hold\n  only annual totals). Limits: the system is chosen for the agent (sizing is T7); tariffs are\n  flat; tool-supplied energy flows mean T8 grades chain execution and not dispatch modelling.\n\nAgents interact through three tools (`src/solarquotebench/tools/`): `pvgis_lookup`,\n`tariff_lookup` and `incentive_schedule` (T6 to T8 also get `usage_profile` and, for T8,\n`energy_flows`), all served from cached fixtures, never\na live API call during grading. The tools return correct raw data on purpose: the\nenvironment tests whether an agent uses the data correctly, not whether it has\nmemorised numbers. `incentive_schedule` is withheld on a subset of T3 instances\nspecifically to separate \"doesn't know the scheme exists\" from \"knows it exists\nbut reasons about it wrongly.\"\n\nEvery grader emits a per-dimension breakdown with tolerance bands, not a single\npass/fail number, and every family's grader can diagnose which known failure\nmode an answer looks like (`diagnosed_error_mode` or equivalent). That\ndiagnosis is what makes the baseline report's failure taxonomy possible; see\n`scripts/run_baseline.py`.\n\n## How ground truth is derived\n\nThe physics is PVGIS (a free public API); the incentive rules are public policy.\nThere is no third-party intellectual property in the maths. What is proprietary\nis SolarSight's own engineering work turning that physics and policy into a\ncorrect, tested pricing engine, and that work is what this environment ports\nand re-verifies rather than re-deriving from scratch.\n\nEach `src/solarquotebench/domain/` module is a direct port of one SolarSight\nTypeScript module, and each carries the same discipline: transcribe the\noriginal test suite's exact asserted values rather than re-deriving new ones,\nso a regression in the port is caught the same way a regression in the\noriginal engine would be. Concretely:\n\n- `domain/pvgis.py` reproduces SolarSight's own cached PVGIS fixtures exactly:\n  `tests/test_pvgis.py::test_pvgis_lookup_reproduces_real_goldcoast_fixture`\n  and `::test_pvgis_lookup_reproduces_real_london_fixture` assert the ported\n  lookup returns the same annual yield SolarSight's real cached fixture\n  recorded, and `::test_compass_to_pvgis_aspect_matches_ts_table` transcribes\n  `pvgis.test.ts`'s own conversion table entry for entry.\n- `domain/incentives.py` transcribes `incentives.test.ts`'s own asserted\n  dollar amounts, not re-derived ones: for example\n  `tests/test_incentives.py::test_au_stc_postcode_derived_rating` asserts the\n  AU federal STC value for a 10.56 kW Gold Coast system comes out to exactly\n  $2,880 (72 certificates at $40), matching the engine's own comment that a\n  naive worked example gives a different, wrong figure. Battery STC tiering,\n  scheme date windows and jurisdiction data-hygiene checks are covered the\n  same way.\n- `domain/finance.py`, `domain/battery.py` and `domain/loadprofile.py` are each\n  marked in their own module docstrings as \"ported and oracle-verified\": direct\n  line-for-line ports of `finance.ts`, `battery.ts` and `loadProfile.ts`, with\n  `tests/test_finance.py`, `tests/test_battery.py` and\n  `tests/test_loadprofile.py` transcribing the original test suite's own\n  assertions (cashflow/NPV/IRR sanity bands, hour-by-hour energy conservation,\n  weekday/weekend and hemisphere-phase load shape determinism).\n- `domain/tariffs.py`'s flat national rates are ported from\n  `countries.ts`/`countries.json` and cross-validated the same way in\n  `tests/test_tariffs.py`. Its time-of-use schedules are not: see the\n  limitations section below.\n\nEvery T1/T4 PVGIS geometry an instance actually uses must already exist as a\ncached fixture (`src/solarquotebench/data/pvgis_fixtures/`); a generator that asks\nfor a geometry nobody pre-fetched raises loudly (`PvgisFixtureError`) rather\nthan silently falling back to an approximation, so a missing fixture fails CI\ninstead of quietly shipping an ungradeable instance.\n\n## What this does not measure, and where ground truth is weaker\n\nThis section exists because a technical buyer should be able to find the\nweak points without digging through source. Everything below is real,\ndocumented in the code it describes, and worth knowing before trusting a\nnumber this environment produces.\n\n**T2's and T4's hourly PV generation shape was authored, and has now been\nchecked against a real oracle (`TASK-V2-ROADMAP.md` R4, 2026-08-20 and\n2026-11-02).** PVGIS's annual endpoint returns only a yearly total, and\nSolarSight's own engine never modelled an hourly series. T2's self-consumption\nobjective and T4's payback/NPV both need an hourly shape to combine with the\nload profile and battery dispatch, so `t2_orientation_allocate/compute.py`\n(and a deliberately separate copy in `t4_tou_payback/compute.py`) builds one\nfrom first-principles solar geometry: a raised-cosine generation bump within\neach day's daylight window, shifted earlier or later by a `sin(aspect)` term\nso an east- or west-facing plane peaks at a different time of day. It is\nanchored so it always sums to exactly the PVGIS-verified annual total, so the\nyearly number is oracle-backed; only the intraday distribution of that total\nis authored. That distribution was compared against PVGIS's real hourly\n`seriescalc` data (`scripts/fetch_pvgis_hourly_fixtures.py`, 5-year averaged,\nlocal standard time, `scripts/analyze_hourly_shape_divergence.py`; numbers in\n`baseline/RESULTS-2026-08-18.md`'s R4 sections). Measured result: T4 NPV\ndiverges a mean 2.2% of system cost (max 6.9%, 13 instances), inside the\ngrader's 20% loose band for every instance. T2's `self_consumption_value`\ndiverges a mean 10.9% (max 21.9%, 7 instances); one three-plane Toronto\ninstance sits just over 20%, and no other is over 16%. A first measurement on\n2026-08-20 reported a 72% divergence for southern-hemisphere sites. That was\na bug in the fixtures (PVGIS returns UTC hours and they were stored as local\ntime), not in the shape model; the model is unchanged and no expected value\nmoved. The shape still does not model cloud clustering or the\ntilt-dependent diffuse/direct split, and its two free parameters (a 4.5-hour\npeak-offset amplitude and a 1.5 sharpness exponent) were chosen empirically,\nnot tuned per instance. Anything resting on this shape is deterministic and\nnow empirically checked at the graded-quantity level, but still not\nphysically validated the way T1's PVGIS-anchored annual totals are. T2's and\nT4's tolerance bands are set wider than T1's for this reason.\n\n**Time-of-use tariff coverage is real for two jurisdictions and synthetic for\nfour.** `domain/tariffs.py` ships a sourced, dated TOU rate table for exactly\nCA (Ontario OEB) and GB (Octopus Economy 7): these were the two jurisdictions\nwhere a citable numeric rate table could actually be retrieved. AU, IE, NZ and\nUS T4 instances instead carry a synthetic intraday rate shape, clearly labelled\nas such on the instance (`TariffSpec.is_synthetic`) and shown directly to the\nagent rather than served through `tariff_lookup` (which would return that\njurisdiction's real flat rate instead, a deliberately visible mismatch, not a\nsilent one). Only the shape, meaning which hours are expensive, is invented;\nthe export price and daily supply charge on a synthetic instance are still the\nreal, sourced flat-rate figures for that jurisdiction. Real schedules live in\n`data/tariffs/tou-schedules.json`; synthetic shapes live only inside the\ninstances that use them and never in that file, so nothing synthetic can be\nmistaken for sourced data by grepping the data directory.\n\n**The T5 seed case does not reproduce the real vendor number exactly, and that\ngap is disclosed rather than smoothed over.** TASK-V1-ENVIRONMENT.md names the\nreal case this family was built to catch: a vendor PDF claiming 19,961 kWh/yr\nfor a 10.56 kW Gold Coast system, against a correct figure of roughly 16,314,\nattributed to the vendor assuming 100% efficiency. The correct figure is\nverified against SolarSight's own PVGIS fixture (matching T1's own reference\ninstance almost exactly). But 19,961 kWh/yr implies about 1,890 kWh/kWp, which\nis roughly 5% above even a zero-loss PVGIS result for that geometry (about\n18,970 kWh/yr). A 100%-efficiency assumption alone cannot explain the real\nvendor's number; the real case evidently involved more than one compounding\nerror. The seeded T5 instance therefore uses the engine-derived 18,970 figure\nas the \"100% efficiency\" wrong value, which keeps the planted error a clean,\nsingle-mechanism `losses_ignored` case rather than an unexplained one. This is\na point in the environment's favour, not a flaw: the discrepancy was found and\nwritten down (`t5_proposal_audit/generator.py`'s own comment on the seed\ninstance) instead of quietly forcing the number to match the anecdote.\n\n**The Canada T3 instance now has jurisdiction-specific cross-validation**\n(2026-08-20, `TASK-V2-ROADMAP.md` R6). AU, US, GB and IE T3 instances each\ntrace to a specific asserted value in SolarSight's own `incentives.test.ts`;\nthe Canada instance (`t3-014-ca-grant-closed-today`) used to be the\nexception, checked only against this repo's own\n`tests/test_incentives.py::test_ca_greener_homes_grant_always_closed_today`.\nThat gap is closed: SolarSight's `incentives.test.ts` gained a\n`ca-federal-greener-homes-grant` worked example\n(github.com/daviddigital/solarsight PR #147, merged as commit `daadbe0` on\n`main`), the same real federal Greener Homes Grant rates (CAD 1,000/kW,\ncapped at CAD 5,000) and eligibility window (2021-05-01 to 2024-02-28) this\ninstance is built from. AU, US, GB, IE and CA are now all independently\ncross-validated against a specific line in the original TypeScript suite.\n\n**The consolidated cross-validation gate is `tests/test_crossvalidation.py`**\n(added 2026-08-20, `TASK-V2-ROADMAP.md` R5). Each `domain/` module still\ncross-validates against its own SolarSight test file independently (see the\nsection above); this file is the single, runnable demonstration\nTASK-V1-ENVIRONMENT.md's non-negotiables call for, in one place: PVGIS's real\nGold Coast fixture, the finance golden-sanity-band and replacement-cost\nassertions, battery hourly energy conservation, and four incentives worked\nexamples (AU STC, AU battery STC tiers, IE SEAI tiers, the US federal\ncredit's 2025/2026 boundary). It imports the same test functions and named\ncase lists the individual module test files already define rather than\nre-typing any pinned number, so there is exactly one source of truth per\nvalue and no way for the two to silently drift apart.\n\nIf anything else in the codebase's own docstrings reads as a limitation, treat\nit as one: `compute.py` and `domain/*.py` modules use phrases like \"newly\nauthored,\" \"no oracle,\" \"genuine modelling simplification\" and \"could not\nretrieve\" precisely so this section does not have to be the only place they\nare said out loud.\n\n## How to run it\n\n### From the Prime Intellect Environments Hub\n\n```\nprime env install solar-sight/solarquotebench\nuv run vf-eval solarquotebench\n```\n\nPushed 2026-08-20 under the `solar-sight` namespace, the Prime team this\nrepo's owner administers, not a personal username: `prime whoami` shows this\naccount as a Team account with no personal username set, and the push uses\nthe team slug instead. Listing: https://app.primeintellect.ai/dashboard/environments/solar-sight/solarquotebench.\n`vf-eval` runs the entry point this repo exposes,\n`solarquotebench.env:load_environment` (see \"`verifiers` entry point\" below), and\nneeds the `verifiers` extra, which `prime env install` pulls in for you.\n\n### From source\n\nRequires Python 3.12+.\n\n```\npython3.12 -m venv .venv\n.venv/bin/pip install -e '.[dev]'\n.venv/bin/pytest -q            # fast suite (excludes slow/exhaustive-search re-derivations)\n.venv/bin/pytest -q -m \"\"      # full suite, including slow tests; what the pre-commit hook runs\n.venv/bin/ruff check .\n```\n\nEach task family's curated instances are available directly from its\ngenerator, e.g. `solarquotebench.tasks.t1_yield_estimate.generator.curated_instances()`.\n\n### Public/private splits\n\n`scripts/export_splits.py` writes one JSONL file per family per split,\n`splits/<public|private>/<family>.jsonl`. Every line is\n`{\"id\": ..., \"prompt_payload\": {...}, \"ground_truth\": {...}}`: `prompt_payload`\nis exactly what that family's `Instance.prompt_payload()` returns, and\n`ground_truth` is the full `Instance.to_dict()`. The two are kept as separate\ntop-level keys, not merged, so a harness wiring an agent up to\n`record[\"prompt_payload\"]` cannot accidentally serve a ground-truth field even\nwithout reading this paragraph; `tests/test_export_splits.py` asserts no\nground-truth key ever leaks into `prompt_payload` for any exported instance,\nacross every family.\n\nPublic splits are `curated_instances()`, the hand-checked set. Private splits\nare `bulk_generate(seed=90210, count=...)`, drawn from the same generator with\na disjoint seed and never individually hand-verified, only structurally\nvalidated (every generated instance runs through the same ground-truth\npipeline the curated set uses). The private split is gitignored\n(`splits/private/`) and never committed or shipped; only the public split goes\nout. Current counts (public / private):\n\n| Family | Public | Private |\n| --- | --- | --- |\n| T1 yield-estimate | 16 | 200 |\n| T2 orientation-allocate | 14 | 80 |\n| T3 incentive-eligibility | 16 | 200 |\n| T4 tou-payback | 13 | 200 |\n| T5 proposal-audit | 16 | 200 |\n\nT2's private count is smaller because each site costs a full exhaustive\nself-consumption search (roughly 1 to 3 seconds); every other family is\nsub-100-millisecond per instance.\n\n### `verifiers` entry point\n\n`src/solarquotebench/env.py` exposes all eight families as a `vf.EnvGroup` of\n`ToolEnv`s (nine sub-environments, not eight: T3's tool-withholding trap needs\ntwo sibling sub-environments, `t3_incentive_eligibility` and `t3_withheld`,\nbecause the installed `verifiers` version advertises one fixed tool list per\nenvironment rather than a per-row dynamic one). It is an optional extra, not\npart of the default install, because `verifiers` pulls in a roughly\n70-package transitive dependency tree that has no reason to sit in every\ncontributor's plain `pytest`/`ruff` loop:\n\n```\n.venv/bin/pip install -e '.[verifiers]'\n```\n\nThis was validated against the real `verifiers==0.3.0` package installed from\nPyPI into a throwaway venv, not written from a remembered interface: the\nactual installed source was read for every shape `env.py` depends on\n(`Environment`, `ToolEnv`, `StatefulToolEnv`, `Rubric`, `EnvGroup`,\n`load_environment`), then `solarquotebench` itself was installed into that same\nvenv and run end to end against it (dataset construction, tool wrappers, and\n`Rubric.score_rollout` against hand-built rollouts for all five families).\n`tests/test_env.py` starts every test with `pytest.importorskip(\"verifiers\")`,\nso it skips cleanly (not silently) in this repo's own `.venv`, where the\nextra is not installed.\n\nOne thing this has NOT verified: a real `Environment.rollout()` against a\nlive LLM endpoint. This repo is the grader side of the RL loop, not model\ninference, and no model endpoint was available to call it against. That step\nis out of scope here and should be run once by anyone integrating this\nenvironment into `prime-rl` or the Environments Hub before trusting it in\nproduction. See `env.py`'s own module docstring for the full research trail.\n\nThe baseline runner, `scripts/run_baseline.py`, drives one or more model\nclients through every family's curated instances, tool calls included, and\ngrades every rollout with that family's own deterministic grader:\n\n```\n.venv/bin/python scripts/run_baseline.py                          # dry run, no network calls\n.venv/bin/python scripts/run_baseline.py --limit 3                # fast smoke test\n.venv/bin/python scripts/run_baseline.py --live --confirm-spend   # the real thing, costs money\n```\n\nIt defaults to `--dry-run` (a deterministic stub model, zero network calls, zero\ncost) and refuses to make a single paid request unless both `--live` and\n`--confirm-spend` are passed; the cost estimate is always printed first, and it\nnever shows a made-up dollar figure for a model this repo has no sourced\npricing for. An unpriced model prints as unknown instead; see `baseline/costs.py`\nand the `--pricing-file` flag if you want a real number.\n\nVendor status as of 2026-08-20, checked live from this machine except where\nstated:\n\n- **OpenAI is confirmed working.** `gpt-5.6-luna`, the current frontier tier on\n  the account behind `OPENAI_API_KEY`, returns a correct completion. It is the\n  default model for `--openai-model`.\n- **Anthropic is implemented but not yet usable.** `ANTHROPIC_API_KEY` is not\n  set anywhere on this machine, and there is no default model id: nothing here\n  could verify a Claude model id against a live endpoint without the key.\n  Set `ANTHROPIC_API_KEY` and pass `--anthropic-model` to use it.\n- **Google Gemini is unreachable from this machine.** `GET /v1beta/models`\n  returns a location error, and every current Gemini model id tried 404s. The\n  client is left in place for a location where the API is reachable, but a\n  `--live` run from here will not get results from it.\n- **Prime Intellect Inference is implemented and confirmed reachable, but\n  will not be run.** `PRIME_API_KEY` is set and works: `prime inference\n  models` lists the full catalog, and a live smoke call reached Prime's API.\n  `_post_json` sends a real User-Agent because Cloudflare in front of\n  `api.pinference.ai` returns `HTTP 403 error code: 1010` for urllib's\n  default (see `baseline/vendors.py`). Both wallets show $0.00 and the owner\n  has declined to fund them, so every call returns `402 insufficient_funds`.\n  Like Anthropic, there is no default model id: pass `--prime-model`.\n- **Claude Code (owner's Claude subscription) is the second vendor, and it\n  ran.** `--vendors claude-code` drives headless `claude -p` with this\n  environment's three tools served by a local MCP server\n  (`baseline/claude_code.py`, `baseline/mcp_server.py`). It needs no API key,\n  only a logged-in `claude` CLI or `CLAUDE_CODE_OAUTH_TOKEN`. This is Claude\n  Code's own agent loop, not the raw API loop the OpenAI client uses: see\n  \"What the baseline shows\" for the disclosure. Model id: `--claude-code-model`\n  (default `claude-sonnet-5-5`).\n\nBecause of this, a `--live` run does not need every vendor to be usable: a\nvendor missing a key (or, for Anthropic and Prime, a model id) is skipped with\na printed reason, and the run continues with whatever vendors it does have,\nselectable with `--vendors openai,anthropic,gemini,prime` (the default, all\nfour). The task spec calls for results from at least two different vendors\nbefore a baseline counts as evidence. A run that ends up with fewer says so\nprominently in its own report, both at the top and in the closing summary,\nrather than reading as if it met that bar.\n\nAPI keys are read from `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` / `GEMINI_API_KEY`\n/ `PRIME_API_KEY` in the environment or from `~/.codylabs/secrets.env`, never\nhardcoded, never logged. Results write incrementally to\n`baseline/results/<run-id>/rollouts.jsonl`, so a crashed or rate-limited run\ncan be resumed by re-running the same command.\n\n## Provenance and licensing posture\n\nThe physics (PVGIS) is a free public API; the incentive rules are public\ngovernment policy. Neither carries third-party intellectual property this\nenvironment needs to protect or license. SolarSight's own engine was used only\nas the verification oracle during development, transcribed test assertions and\nall; nothing in this repository imports or depends on SolarSight's code at\nruntime, per `TASK-V1-ENVIRONMENT.md`'s non-negotiables.\n\nThis repository is licensed under the MIT License (see `LICENSE`). That\ncovers everything shipped here, meaning the code and the public split's\nground truth; `splits/private/`, the held-out set, never ships at all (see\n\"Public/private splits\" above and `tests/test_packaging.py`), so the licence\nquestion does not arise for it.\n\n## What the baseline shows\n\nTwo vendors have been run on all 75 curated public instances, zero crashed and\nzero truncated in both:\n\n- `luna-final`, 2026-08-18: OpenAI `gpt-5.6-luna`, reasoning effort medium,\n  through this repo's raw API loop.\n- `claude-sonnet-5-5-r1`, 2026-11-02: Anthropic `claude-sonnet-5-5` through\n  Claude Code headless (`claude -p`), no `--effort` flag so Claude Code's\n  default applies.\n\n**Harness disclosure.** The Claude run uses Claude Code's own agent loop\n(its own tool-call plumbing, context handling and turn accounting), not the\nraw API request loop `OpenAIClient` used for the OpenAI run. A score gap\nbetween the two can come from the harness as well as the model. For an\napples-to-apples reproduction with a comparable harness, use `AnthropicClient`\nin `baseline/vendors.py` with your own `ANTHROPIC_API_KEY`. The Claude run was\nisolated: only the three MCP tools were available, no built-in tools, no\nother MCP servers, an empty working directory, and no CLAUDE.md or memory\nloaded (verified on every rollout's init event). The account email and\ntoday's date still reach the context, and a fixed \"Claude agent\" prefix\nprecedes the system prompt; details in `baseline/RESULTS-2026-08-18.md`.\n\nBoth runs under the current graders (`luna-final` regraded for the R3 T2\nweights with `scripts/regrade_run.py`; no model calls):\n\n| Family | luna mean / solved | Claude Sonnet 5.5 mean / solved |\n| --- | --- | --- |\n| T1 yield-estimate | 0.999 / 16 of 16 | 1.000 / 16 of 16 |\n| T2 orientation-allocate | 0.765 / 9 of 14 (0.922 under the pre-R3 weights) | 0.765 / 9 of 14 |\n| T3 incentive-eligibility | 0.749 / 8 of 16 | 0.944 / 14 of 16 |\n| T4 tou-payback | 0.379 / 0 of 13 | 0.415 / 1 of 13 |\n| T5 proposal-audit | 0.600 / 3 of 16 | 0.966 / 13 of 16 |\n| Overall | 0.710 | 0.836 |\n\nFull numbers, per-dimension breakdowns, failure taxonomies and divergences are\nin `baseline/RESULTS-2026-08-18.md`. In short: T1 is at ceiling for both\nmodels; T2 is identical for both (the same 5 value-objective instances fail);\nT4 is near the floor for both; T3 and T5 are where the models differ, with the\nClaude run getting the T3 dollar values right and not over-flagging clean T5\nproposals. The T1 ceiling disclosure below stands.\n\n**T1 is a solved control: 16 of 16 solved by both models, means 0.999 and\n1.000, at ceiling, and it does not discriminate.** The spec's own rule applies here: a family scoring\nnear 100% is too easy and worth nothing, and that should be said plainly\nrather than shipped quietly. T1 should be read as a warm-up or a control, not\nas evidence of difficulty, and it is a candidate for being dropped or made\nsubstantially harder. The evidence that the convention traps are real, not\nfake difficulty, is the reasoning-on-versus-off contrast: the same model,\n`gpt-5.6-luna`, with reasoning disabled falls straight into the\naspect-conversion trap and scores 0.55 on the same task design (an earlier\nsmoke test, `baseline/results/smoke1/`). With reasoning on, medium effort, it\nsolves all 16. So the traps genuinely catch a non-reasoning model; they are\njust not hard for a frontier model that is allowed to reason.\n\nT2's allocation solver is also effectively solved (value_fraction 0.989,\nfeasible 1.000); the family's real signal is all in distinguishes_objectives,\nat 0.643 (and 0 of 5 on the instances that can actually test it: control sites\nand yield-objective instances cannot fail this dimension by construction). The\ngrader has been reweighted to reflect that (`baseline/RESULTS-2026-08-18.md`'s\nR3 section, 2026-08-20, `pyproject.toml` 0.2.0): value_fraction/feasible/\ndistinguishes_objectives moved from 0.6/0.2/0.2 to 0.25/0.10/0.65, derived\nfrom each dimension's measured spread in `luna-final`, not chosen a priori.\nUnder the new weights `luna-final`'s T2 mean is 0.765, not 0.922; the 0.922\nfigure was flattering the family exactly as predicted. That R3 scheme was itself\nsuperseded in 0.4.0 (V3 item A, grader version 3): the binary dimension penalised\nthree of five value instances where the max-yield answer is within 1.6% of the\noptimum, so the score is now regret-based (0.90 value score, 0.10 feasible), with\n`distinguishes_objectives` kept as a diagnostic that only counts where the max-yield\nanswer loses at least 5%. Both baselines restate to T2 0.863 (section \"V3 item A\" of\nthe results doc). T3 identifies which incentive\nschemes apply almost perfectly (0.938 precision and recall) but computes\ntheir value poorly (0.504), which is a different failure than the spec\nexpected and worth noting as such. T5 is the strongest result: it finds\nevery planted error (recall 1.000) but over-flags clean proposals\n(precision 0.517), which is exactly the calibration failure it was built to\nmeasure.\n\nT4's 0.379 is still the family with the weakest oracle behind it: its ground\ntruth depends partly on an hourly generation shape this repo authored, now\nmeasured against PVGIS's real hourly data rather than unvalidated (see the\nlimitations section above): a mean 2.2% NPV divergence across\nall 13 T4 instances. Its npv_accuracy of\n0.091 has been audited, though (`baseline/RESULTS-2026-08-18.md`'s R2\nsection, 2026-08-20): the tolerance band is not too tight. In all 13 T4\nrollouts the agent's NPV error was larger than the smallest known\nwrong-formula decoy error for that same instance, so the low score reflects\ngenuine multi-step arithmetic failure, not an unfairly tight band. The\nprompt payload's wording was clarified to state its discounting/escalation/\ndegradation conventions explicitly, closing a minor under-specification gap\nthat audit surfaced, but that fix is too small to explain the measured\nscore. Full reasoning for all five families is in\n`baseline/RESULTS-2026-08-18.md`.\n\nThe overall mean across all 75 instances, regraded under the R3 T2 weights,\nis 0.710 for luna (0.739 under the pre-R3 weights) and 0.836 for the Claude\nrun. Do not read that number on its own:\nit hides T1's ceiling, and it should not be quoted without the per-family\ntable above.\n\n## Status\n\nTwo baseline runs exist on the same 75 curated instances from two vendors:\n`luna-final` (OpenAI, 2026-08-18) and `claude-sonnet-5-5-r1` (Anthropic via\nClaude Code, 2026-11-02). The task spec's two-vendor bar is met. The two runs\nuse different harnesses (see the disclosure under \"What the baseline shows\"),\nso read gaps between them as model plus harness, not model alone. Full detail\nis in `baseline/RESULTS-2026-08-18.md`.\n\nNot run: Gemini (geo-blocked from this machine), the raw-API Anthropic path\n(no `ANTHROPIC_API_KEY`), and Prime Inference (wallet unfunded, owner declined).\nAll three clients stay in `baseline/vendors.py` for anyone with access.\n\n### V3 (0.4.0): what was added and where it stands\n\n`TASK-V3-FULL-SCOPE.md` is the spec and status log. T2 got a regret-based value score\n(grader version 3), and T6 to T8 were added. Public/private counts: T6 16/200, T7 14/24,\nT8 14/60 (T7 is slow to generate because every instance is solved by exhaustive\nenumeration, about 7 seconds each). Claude Sonnet 5.5 via Claude Code scores T6 1.000,\nT7 0.771, T8 0.987 (`baseline/RESULTS-2026-08-18.md`, section \"V3\"): T6 and T8 are at\nceiling for that model with the tools offered, T7 has headroom (time-of-use and\nmin-cost-with-EV instances). `gpt-5.6-luna` (reasoning effort medium) scores T6 0.944, T7 0.591, T8 0.916 on the same instances\n(rebates are its weak dimension: it returned a zero Australian STC on all four Gold Coast T8\ninstances); the side-by-side tables are in that section. Open\nlevers: stricter hardened tiers (see below), recurating T2's three\nnon-material value instances to at least 5% divergence, a published dataset for\nhousehold-profile consumption estimates.\n\n### Hardened control tiers T6H and T8H (0.5.0)\n\n`t6h_consumption_hard` and `t8h_full_quote_hard` are harder versions of T6 and T8, added next to\nthem with the originals untouched. Ship bar: Sonnet 5.5 mean below about 0.9 with genuine failures.\nNeither tier met it, so both are labelled control tiers: they are not in the default environment,\nsplits or listing, and live in `splits/hardened-control/` (public committed, private not). Run them\nwith `scripts/run_baseline.py --families t6h,t8h`.\n\n- T6H (12 public / 120 private): unprinted tax (AU and NZ only), post-tax credit, demand charge,\n  reissued bill, estimated then actual read, plan change, 15 months of history, a missing bill,\n  solar export. Sonnet 5.5: 1.000 on two different sets of 12.\n- T8H (12 public / 60 private): messy enquiry thread with corrections, time-of-use tariffs\n  (`energy_flows` takes optional `tou_windows`), install-year, postcode-zone and battery-window\n  rebates, loss and projection assumptions stated in the thread. Sonnet 5.5: 0.978 (10 of 12). A\n  first version scored 0.881 but three defects in our own task text (an unstated STC deeming rule, an\n  install month before the battery rebate opens, a contradicting EV note) caused most misses, so it\n  was fixed and rerun, see `baseline/RESULTS-2026-08-18.md`, section \"Hardened tiers\".\n- Honest reading: these show that execution of a well specified chain is reliable for the strongest\n  model; the remaining headroom is rule recall (US credit not upfront, battery tier arithmetic). Luna\n  on the hardened tiers is pending owner approval; the command and token estimate are in the results\n  doc.\n\n## TODO\n\nEverything outstanding, grouped by kind. This section is the single place to\nlook for outstanding work in this repo, per the portfolio convention: no\nseparate `TODO.md`. The detailed spec for all outstanding work, in priority\norder, is `TASK-V2-ROADMAP.md`; read it before picking up any item below.\n\n### Done\n\n1. Prime Intellect account created, `prime login` completed. The account is\n   a Team account (`SolarSight`, slug `solar-sight`) with no personal\n   username set; the team slug is the publishing namespace instead, and\n   nothing here needed a personal username to work.\n\n2. Prime API key created with the Inference permission and stored as\n   `PRIME_API_KEY` in `~/.codylabs/secrets.env`. Confirmed working live:\n   `prime inference models` lists the full catalog.\n\n3. Pushed to the Environments Hub with `prime env push`, public visibility:\n   https://app.primeintellect.ai/dashboard/environments/solar-sight/solarquotebench.\n   The `<owner>` placeholder in the README's Hub install instructions above\n   is replaced with the real namespace, `solar-sight`. Re-pushed 2026-08-20\n   (`TASK-V2-ROADMAP.md` R8, `prime env push --auto-bump`, now v0.2.1) after\n   finding the live listing was stale and rendering the pre-R2-through-R7\n   README; also caught and fixed a real bug in that CLI command, which\n   rewrote `[tool.ruff] target-version` from `\"py312\"` to the new package\n   version (it matches any TOML key containing \"version\", not only\n   `[project] version`). Check the sheet linked from\n   primeintellect.ai/blog/scaling-environments-program's \"Browse Open\n   Bounties\" button periodically for the bounty program reopening: both\n   tracks (Open-Access and Application-Only) were checked and explicitly\n   marked closed as of this date, and neither's listed categories fit a\n   novel domain environment like this one anyway.\n\n### Blocked on the owner\n\n4. Confirm or change the MIT licence. It was a default choice, not a\n   considered decision. Rationale recorded here: the public split going out\n   permissively is consistent with the spec, and the 880-instance private\n   held-out split is the actual moat and never ships.\n\n5. CANCELLED (2026-08-20). Funding the Prime wallet. The owner declined every\n   paid second-vendor path (Anthropic API key, Gemini, Prime wallet deposit).\n   Prime Inference stays in the repo as a built and reachable client\n   (`PrimeClient` in `baseline/vendors.py`), but nothing here will fund it.\n\n### Close the spec's last definition of done item\n\n6. DONE (2026-11-02, `TASK-V2-ROADMAP.md` R1). The two-vendor bar is met:\n   Claude Sonnet 5.5 ran all 75 instances through headless `claude -p` with\n   the tools served by a local MCP server, next to `luna-final`. The harness\n   difference is disclosed in \"What the baseline shows\" and in\n   `baseline/RESULTS-2026-08-18.md`. The Hub listing was re-pushed\n   on 2026-11-02 under the new name (see item 13), so its README now matches\n   (R1 step 9 done).\n\n### Product decisions raised by the baseline\n\n7. T1 is at ceiling: 16 of 16 solved by both `gpt-5.6-luna` (0.999) with\n   reasoning on and Claude Sonnet 5.5 (1.000). The spec says a family at ceiling is worth nothing. It\n   currently ships labelled as a solved control. Decide whether to harden it\n   (harder geometries, compounding multi-plane traps, traps that survive\n   reasoning) or drop it. Evidence that the traps are real but easy: the\n   same model with reasoning disabled scored 0.55 and fell into the\n   aspect-conversion trap.\n\n8. DONE (2026-08-20, `TASK-V2-ROADMAP.md` R3). T2 was half at ceiling:\n   `value_fraction` 0.989 and `feasible` 1.000 meant the allocation\n   optimisation was solved, while all the discriminating signal was in\n   `distinguishes_objectives` at 0.643 (0 of 5 on the instances that can\n   actually test it). Reweighted the grader from that measured spread:\n   0.6/0.2/0.2 to 0.25/0.10/0.65\n   (value_fraction/feasible/distinguishes_objectives). `luna-final`'s T2\n   mean under the new weights is 0.765, not 0.922. See\n   `baseline/RESULTS-2026-08-18.md`'s R3 section and\n   `scripts/regrade_t2_reweight.py`, the committed regrade script.\n\n### Technical follow-ups\n\n9. DONE (2026-08-20, `TASK-V2-ROADMAP.md` R2). T4's `npv_accuracy` is 0.091\n   with 0 of 13 solved. Audited whether the NPV tolerance band was simply\n   too tight: it is not. In all 13 rollouts the agent's NPV error exceeded\n   the smallest known wrong-formula decoy error for that instance, so the\n   score reflects genuine arithmetic failure. The prompt payload's wording\n   was clarified to state the discounting/escalation/degradation\n   conventions explicitly, a minor under-specification fix, not a scoring\n   change. See `baseline/RESULTS-2026-08-18.md`'s R2 section. T4's ground\n   truth still rests partly on the newly authored hourly PV shape that has\n   no oracle (see item 12 below), which is a separate, still-open question\n   from the tolerance band.\n\n10. DONE (2026-08-20, `TASK-V2-ROADMAP.md` R5). Added\n    `tests/test_crossvalidation.py`, importing the same test functions and\n    case lists each domain module's own test file already defines, as the\n    single runnable demonstration of the spec's cross-validation\n    non-negotiable.\n\n11. DONE (2026-08-20, `TASK-V2-ROADMAP.md` R6). The T3 Canada instance had\n    real ported eligibility dates but no jurisdiction-specific assertion in\n    SolarSight's TypeScript suite to cross-validate against, unlike AU, US,\n    GB and IE. Fixed upstream: `incentives.test.ts` gained a\n    `ca-federal-greener-homes-grant` worked example\n    (github.com/daviddigital/solarsight PR #147, merged).\n\n12. DONE (2026-11-02, `TASK-V2-ROADMAP.md` R4). Measured T2/T4's shared\n    hourly PV shape against PVGIS's real hourly data. The first measurement\n    (2026-08-20) showed southern-hemisphere instances 72% off; the cause was\n    the oracle fixtures being stored in UTC rather than local standard time,\n    not the shape model. With the fixtures corrected, T4 NPV diverges a mean\n    2.2% of system cost (max 6.9%) and T2 `self_consumption_value` a mean\n    10.9% (max 21.9%). No expected value changed. See\n    `baseline/RESULTS-2026-08-18.md`'s \"R4 phase 2\" section.\n\n13. DONE (2026-11-02). Project renamed from solar-eval to SolarQuoteBench,\n    version 0.3.0. The `prime` CLI was upgraded from 0.6.25 to 0.19.0 (the\n    self-upgrade failed because `uv` was not on PATH; `uv tool upgrade prime`\n    via a temporary uv install worked). The environment was pushed public as\n    `solar-sight/solarquotebench`, a new listing (new environment id), using\n    `prime env push --team solar-sight --visibility PUBLIC` (`--owner` fails\n    for a first push). Listing:\n    https://app.primeintellect.ai/dashboard/environments/solar-sight/solarquotebench\n    Install check: a throwaway venv install from the Hub index followed by\n    `load_environment()` with no arguments built the 75-instance EnvGroup. The\n    old listing `solar-sight/solar-eval` was kept and got one final version\n    (0.2.3, code unchanged) whose README points to the new listing. The\n    0.19.0 push did not touch `[tool.ruff] target-version` (still \"py312\").\n    The Hub lists the package as Unclassified because `pyproject.toml` has no\n    verifiers requirement; pass `--runtime v0|v1` on the next push if a\n    classification is wanted.","encoding":"utf-8","truncated":false,"total_bytes":44585},"status":null}