{"data":{"kind":"file","path":"README.md","version_id":"lkdgw5buyu5vvd19v1v99ggp","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":37383,"modified_at":"2026-08-20T08:27:06.990000","content_hash":"312a1e88c34add2a39b505b7cd274bbe3c314923750b403be3a94e58c337004f"},"entries":[],"content":"# solar-eval\n\nA verifiable-reward RL environment for residential solar and battery quoting. An\nagent is given a site, a system spec, a tariff, or a finished vendor proposal, and\nmust return a numeric answer or a structured audit. Every answer is graded by a\ndeterministic Python oracle, never an LLM judge. The full spec and design\nrationale is in `TASK-V1-ENVIRONMENT.md`; this file is the front door for anyone\ndeciding whether to run or buy it.\n\nThe environment is Python, targeting `verifiers`/`prime-rl`-style harnesses (this\nmarket is Python-only; see `TASK-V1-ENVIRONMENT.md`'s non-negotiables). The\ngrading engine is ported from a real production solar-quoting system,\n[SolarSight](../solarsight), whose own TypeScript test suite is the oracle used to\nverify the port. SolarSight's code is not a runtime dependency of anything here.\n\n## What it measures\n\nFive task families, `src/solar_eval/tasks/t1_yield_estimate` through\n`t5_proposal_audit`. Each has its own generator, grader and at least twelve\nhand-checked instances (`generator.py`'s `curated_instances()`).\n\n- **T1 yield-estimate** (solved control, not a discriminating family): given\n  a site, roof planes and a panel spec, return annual kWh. The value is in\n  three convention traps: compass bearing versus PVGIS's own south-zero\n  aspect convention (a sign error that produces a plausible-looking wrong\n  number, not a crash), hemisphere-dependent optimal tilt, and multiplicative\n  (not additive) composition of system losses. State this plainly up front:\n  against `gpt-5.6-luna` with reasoning on, T1 scores 0.999, 16 of 16 solved,\n  which is ceiling and does not discriminate between models. The traps are\n  real, not fake difficulty: the same model with reasoning turned off falls\n  into the aspect-conversion trap and scores 0.55. So T1 measures whether a\n  reasoning model avoids these traps, which frontier models with reasoning\n  enabled already do; treat it as a control and a regression check, not as\n  evidence of difficulty. Full numbers are in \"What the baseline shows\" below.\n- **T2 orientation-allocate**: allocate a fixed panel budget across multiple\n  roof planes. Every site ships as a matched pair, one instance maximising raw\n  annual yield, the other maximising the dollar value of self-consumption under\n  a tariff and load profile. The two objectives genuinely disagree on some\n  sites (a west-facing plane can win on value while losing on yield); an agent\n  that treats them as the same question fails.\n- **T3 incentive-eligibility**: given a jurisdiction, system spec and install\n  date, return which government incentive schemes apply and their combined\n  value. This is the family with the highest expected failure rate, because an\n  LLM's knowledge of incentive schedules is stale by construction: AU's STC\n  deeming schedule and battery rebate, UK MCS/SEG, IE SEAI grant tiers, US ITC,\n  NZ and CA schemes, including cases where the correct answer is \"not\n  eligible\" and cases straddling a rate step-down.\n- **T4 tou-payback**: given a time-of-use tariff, load profile, system and\n  battery spec, and export limit, return simple payback year and 25-year NPV,\n  graded as two independent dimensions. Tests multi-step arithmetic under price\n  escalation, degradation, export caps and self-consumption timing.\n- **T5 proposal-audit**: given a complete, professional-looking vendor\n  proposal, some clean and some carrying planted errors of a known type and\n  magnitude, find, classify and quantify each error. This is the most novel\n  family: it measures deference to an authoritative-looking wrong number, not\n  raw computation, and clean instances are just as important as dirty ones\n  because a model that flags everything indiscriminately must score badly on\n  them.\n\nAgents interact through three tools (`src/solar_eval/tools/`): `pvgis_lookup`,\n`tariff_lookup` and `incentive_schedule`, all served from cached fixtures, never\na live API call during grading. The tools return correct raw data on purpose: the\nenvironment tests whether an agent uses the data correctly, not whether it has\nmemorised numbers. `incentive_schedule` is withheld on a subset of T3 instances\nspecifically to separate \"doesn't know the scheme exists\" from \"knows it exists\nbut reasons about it wrongly.\"\n\nEvery grader emits a per-dimension breakdown with tolerance bands, not a single\npass/fail number, and every family's grader can diagnose which known failure\nmode an answer looks like (`diagnosed_error_mode` or equivalent). That\ndiagnosis is what makes the baseline report's failure taxonomy possible; see\n`scripts/run_baseline.py`.\n\n## How ground truth is derived\n\nThe physics is PVGIS (a free public API); the incentive rules are public policy.\nThere is no third-party intellectual property in the maths. What is proprietary\nis SolarSight's own engineering work turning that physics and policy into a\ncorrect, tested pricing engine, and that work is what this environment ports\nand re-verifies rather than re-deriving from scratch.\n\nEach `src/solar_eval/domain/` module is a direct port of one SolarSight\nTypeScript module, and each carries the same discipline: transcribe the\noriginal test suite's exact asserted values rather than re-deriving new ones,\nso a regression in the port is caught the same way a regression in the\noriginal engine would be. Concretely:\n\n- `domain/pvgis.py` reproduces SolarSight's own cached PVGIS fixtures exactly:\n  `tests/test_pvgis.py::test_pvgis_lookup_reproduces_real_goldcoast_fixture`\n  and `::test_pvgis_lookup_reproduces_real_london_fixture` assert the ported\n  lookup returns the same annual yield SolarSight's real cached fixture\n  recorded, and `::test_compass_to_pvgis_aspect_matches_ts_table` transcribes\n  `pvgis.test.ts`'s own conversion table entry for entry.\n- `domain/incentives.py` transcribes `incentives.test.ts`'s own asserted\n  dollar amounts, not re-derived ones: for example\n  `tests/test_incentives.py::test_au_stc_postcode_derived_rating` asserts the\n  AU federal STC value for a 10.56 kW Gold Coast system comes out to exactly\n  $2,880 (72 certificates at $40), matching the engine's own comment that a\n  naive worked example gives a different, wrong figure. Battery STC tiering,\n  scheme date windows and jurisdiction data-hygiene checks are covered the\n  same way.\n- `domain/finance.py`, `domain/battery.py` and `domain/loadprofile.py` are each\n  marked in their own module docstrings as \"ported and oracle-verified\": direct\n  line-for-line ports of `finance.ts`, `battery.ts` and `loadProfile.ts`, with\n  `tests/test_finance.py`, `tests/test_battery.py` and\n  `tests/test_loadprofile.py` transcribing the original test suite's own\n  assertions (cashflow/NPV/IRR sanity bands, hour-by-hour energy conservation,\n  weekday/weekend and hemisphere-phase load shape determinism).\n- `domain/tariffs.py`'s flat national rates are ported from\n  `countries.ts`/`countries.json` and cross-validated the same way in\n  `tests/test_tariffs.py`. Its time-of-use schedules are not: see the\n  limitations section below.\n\nEvery T1/T4 PVGIS geometry an instance actually uses must already exist as a\ncached fixture (`src/solar_eval/data/pvgis_fixtures/`); a generator that asks\nfor a geometry nobody pre-fetched raises loudly (`PvgisFixtureError`) rather\nthan silently falling back to an approximation, so a missing fixture fails CI\ninstead of quietly shipping an ungradeable instance.\n\n## What this does not measure, and where ground truth is weaker\n\nThis section exists because a technical buyer should be able to find the\nweak points without digging through source. Everything below is real,\ndocumented in the code it describes, and worth knowing before trusting a\nnumber this environment produces.\n\n**T2's and T4's hourly PV generation shape has no oracle, and it has now been\nmeasured against one (2026-08-20, `TASK-V2-ROADMAP.md` R4 phase 1).** PVGIS\nonly ever returns an annual total; it has no hourly time series, and\nSolarSight's own engine never modelled one. T2's self-consumption objective\nand T4's payback/NPV both need an hourly shape to combine with the load\nprofile and battery dispatch, so `t2_orientation_allocate/compute.py` (and a\ndeliberately separate copy in `t4_tou_payback/compute.py`) builds one from\nfirst-principles solar geometry: a raised-cosine generation bump within each\nday's daylight window, shifted earlier or later by a `sin(aspect)` term so an\neast- or west-facing plane's generation genuinely peaks at a different time\nof day. It is anchored so it always sums to exactly the PVGIS-verified\nannual total, so the yearly number is still oracle-backed; only the intraday\ndistribution of that total was not, until it was checked against PVGIS's\nreal hourly `seriescalc` endpoint (`scripts/fetch_pvgis_hourly_fixtures.py`,\n5-year averaged, `scripts/analyze_hourly_shape_divergence.py`; full numbers\nin `baseline/RESULTS-2026-08-18.md`'s R4 section). The measured result is\nnot uniform: T4's northern-hemisphere instances (Toronto, London, Phoenix,\nDublin) diverge a mean 6.8% of system cost on NPV, comfortably inside the\ngrader's own 20% loose band; its southern-hemisphere instances (Gold Coast,\nAuckland) diverge a mean 72.2%, far outside it. T2 shows the identical\nsplit on its own graded `self_consumption_value`. Swapping the ground truth\nto the PVGIS-derived shape (a large change: recomputing every T2/T4\nexpected value, re-verification, re-exported splits, a regraded baseline)\nis scoped, evidence-backed future work, not yet done; see the R4 section for\nwhy a full swap would be disproportionate to what the data shows is\nactually wrong (a hemisphere-localized problem, not a uniform one). The\nshape's two free parameters (a 4.5-hour peak-offset amplitude and a 1.5\nsharpness exponent) were chosen empirically to produce a measurable,\ngenuine west-favours-afternoon-self-consumption effect on real cached\ngeometries, not tuned per instance to force a particular answer, and not a\nclaim to reproduce real minute-by-minute solar output (cloud clustering,\ntilt-dependent diffuse/direct split and other real effects are not\nmodelled). Read `t2_orientation_allocate/compute.py`'s module docstring for\nthe full reasoning. Anything resting on this shape, meaning T2's\n`max_self_consumption_value` objective and all of T4, is internally\nconsistent and deterministic, not physically validated the way T1's\nPVGIS-anchored annual totals are. T2's and T4's grading tolerance bands are\nset wider than T1's for exactly this reason.\n\n**Time-of-use tariff coverage is real for two jurisdictions and synthetic for\nfour.** `domain/tariffs.py` ships a sourced, dated TOU rate table for exactly\nCA (Ontario OEB) and GB (Octopus Economy 7): these were the two jurisdictions\nwhere a citable numeric rate table could actually be retrieved. AU, IE, NZ and\nUS T4 instances instead carry a synthetic intraday rate shape, clearly labelled\nas such on the instance (`TariffSpec.is_synthetic`) and shown directly to the\nagent rather than served through `tariff_lookup` (which would return that\njurisdiction's real flat rate instead, a deliberately visible mismatch, not a\nsilent one). Only the shape, meaning which hours are expensive, is invented;\nthe export price and daily supply charge on a synthetic instance are still the\nreal, sourced flat-rate figures for that jurisdiction. Real schedules live in\n`data/tariffs/tou-schedules.json`; synthetic shapes live only inside the\ninstances that use them and never in that file, so nothing synthetic can be\nmistaken for sourced data by grepping the data directory.\n\n**The T5 seed case does not reproduce the real vendor number exactly, and that\ngap is disclosed rather than smoothed over.** TASK-V1-ENVIRONMENT.md names the\nreal case this family was built to catch: a vendor PDF claiming 19,961 kWh/yr\nfor a 10.56 kW Gold Coast system, against a correct figure of roughly 16,314,\nattributed to the vendor assuming 100% efficiency. The correct figure is\nverified against SolarSight's own PVGIS fixture (matching T1's own reference\ninstance almost exactly). But 19,961 kWh/yr implies about 1,890 kWh/kWp, which\nis roughly 5% above even a zero-loss PVGIS result for that geometry (about\n18,970 kWh/yr). A 100%-efficiency assumption alone cannot explain the real\nvendor's number; the real case evidently involved more than one compounding\nerror. The seeded T5 instance therefore uses the engine-derived 18,970 figure\nas the \"100% efficiency\" wrong value, which keeps the planted error a clean,\nsingle-mechanism `losses_ignored` case rather than an unexplained one. This is\na point in the environment's favour, not a flaw: the discrepancy was found and\nwritten down (`t5_proposal_audit/generator.py`'s own comment on the seed\ninstance) instead of quietly forcing the number to match the anecdote.\n\n**The Canada T3 instance now has jurisdiction-specific cross-validation**\n(2026-08-20, `TASK-V2-ROADMAP.md` R6). AU, US, GB and IE T3 instances each\ntrace to a specific asserted value in SolarSight's own `incentives.test.ts`;\nthe Canada instance (`t3-014-ca-grant-closed-today`) used to be the\nexception, checked only against this repo's own\n`tests/test_incentives.py::test_ca_greener_homes_grant_always_closed_today`.\nThat gap is closed: SolarSight's `incentives.test.ts` gained a\n`ca-federal-greener-homes-grant` worked example\n(github.com/daviddigital/solarsight PR #147, merged as commit `daadbe0` on\n`main`), the same real federal Greener Homes Grant rates (CAD 1,000/kW,\ncapped at CAD 5,000) and eligibility window (2021-05-01 to 2024-02-28) this\ninstance is built from. AU, US, GB, IE and CA are now all independently\ncross-validated against a specific line in the original TypeScript suite.\n\n**The consolidated cross-validation gate is `tests/test_crossvalidation.py`**\n(added 2026-08-20, `TASK-V2-ROADMAP.md` R5). Each `domain/` module still\ncross-validates against its own SolarSight test file independently (see the\nsection above); this file is the single, runnable demonstration\nTASK-V1-ENVIRONMENT.md's non-negotiables call for, in one place: PVGIS's real\nGold Coast fixture, the finance golden-sanity-band and replacement-cost\nassertions, battery hourly energy conservation, and four incentives worked\nexamples (AU STC, AU battery STC tiers, IE SEAI tiers, the US federal\ncredit's 2025/2026 boundary). It imports the same test functions and named\ncase lists the individual module test files already define rather than\nre-typing any pinned number, so there is exactly one source of truth per\nvalue and no way for the two to silently drift apart.\n\nIf anything else in the codebase's own docstrings reads as a limitation, treat\nit as one: `compute.py` and `domain/*.py` modules use phrases like \"newly\nauthored,\" \"no oracle,\" \"genuine modelling simplification\" and \"could not\nretrieve\" precisely so this section does not have to be the only place they\nare said out loud.\n\n## How to run it\n\n### From the Prime Intellect Environments Hub\n\n```\nprime env install solar-sight/solar-eval\nuv run vf-eval solar-eval\n```\n\nPushed 2026-08-20 under the `solar-sight` namespace, the Prime team this\nrepo's owner administers, not a personal username: `prime whoami` shows this\naccount as a Team account with no personal username set, and the push uses\nthe team slug instead. Listing: https://app.primeintellect.ai/dashboard/environments/solar-sight/solar-eval.\n`vf-eval` runs the entry point this repo exposes,\n`solar_eval.env:load_environment` (see \"`verifiers` entry point\" below), and\nneeds the `verifiers` extra, which `prime env install` pulls in for you.\n\n### From source\n\nRequires Python 3.12+.\n\n```\npython3.12 -m venv .venv\n.venv/bin/pip install -e '.[dev]'\n.venv/bin/pytest -q            # fast suite (excludes slow/exhaustive-search re-derivations)\n.venv/bin/pytest -q -m \"\"      # full suite, including slow tests; what the pre-commit hook runs\n.venv/bin/ruff check .\n```\n\nEach task family's curated instances are available directly from its\ngenerator, e.g. `solar_eval.tasks.t1_yield_estimate.generator.curated_instances()`.\n\n### Public/private splits\n\n`scripts/export_splits.py` writes one JSONL file per family per split,\n`splits/<public|private>/<family>.jsonl`. Every line is\n`{\"id\": ..., \"prompt_payload\": {...}, \"ground_truth\": {...}}`: `prompt_payload`\nis exactly what that family's `Instance.prompt_payload()` returns, and\n`ground_truth` is the full `Instance.to_dict()`. The two are kept as separate\ntop-level keys, not merged, so a harness wiring an agent up to\n`record[\"prompt_payload\"]` cannot accidentally serve a ground-truth field even\nwithout reading this paragraph; `tests/test_export_splits.py` asserts no\nground-truth key ever leaks into `prompt_payload` for any exported instance,\nacross all five families.\n\nPublic splits are `curated_instances()`, the hand-checked set. Private splits\nare `bulk_generate(seed=90210, count=...)`, drawn from the same generator with\na disjoint seed and never individually hand-verified, only structurally\nvalidated (every generated instance runs through the same ground-truth\npipeline the curated set uses). The private split is gitignored\n(`splits/private/`) and never committed or shipped; only the public split goes\nout. Current counts (public / private):\n\n| Family | Public | Private |\n| --- | --- | --- |\n| T1 yield-estimate | 16 | 200 |\n| T2 orientation-allocate | 14 | 80 |\n| T3 incentive-eligibility | 16 | 200 |\n| T4 tou-payback | 13 | 200 |\n| T5 proposal-audit | 16 | 200 |\n\nT2's private count is smaller because each site costs a full exhaustive\nself-consumption search (roughly 1 to 3 seconds); every other family is\nsub-100-millisecond per instance.\n\n### `verifiers` entry point\n\n`src/solar_eval/env.py` exposes all five families as a `vf.EnvGroup` of\n`ToolEnv`s (six sub-environments, not five: T3's tool-withholding trap needs\ntwo sibling sub-environments, `t3_incentive_eligibility` and `t3_withheld`,\nbecause the installed `verifiers` version advertises one fixed tool list per\nenvironment rather than a per-row dynamic one). It is an optional extra, not\npart of the default install, because `verifiers` pulls in a roughly\n70-package transitive dependency tree that has no reason to sit in every\ncontributor's plain `pytest`/`ruff` loop:\n\n```\n.venv/bin/pip install -e '.[verifiers]'\n```\n\nThis was validated against the real `verifiers==0.3.0` package installed from\nPyPI into a throwaway venv, not written from a remembered interface: the\nactual installed source was read for every shape `env.py` depends on\n(`Environment`, `ToolEnv`, `StatefulToolEnv`, `Rubric`, `EnvGroup`,\n`load_environment`), then `solar-eval` itself was installed into that same\nvenv and run end to end against it (dataset construction, tool wrappers, and\n`Rubric.score_rollout` against hand-built rollouts for all five families).\n`tests/test_env.py` starts every test with `pytest.importorskip(\"verifiers\")`,\nso it skips cleanly (not silently) in this repo's own `.venv`, where the\nextra is not installed.\n\nOne thing this has NOT verified: a real `Environment.rollout()` against a\nlive LLM endpoint. This repo is the grader side of the RL loop, not model\ninference, and no model endpoint was available to call it against. That step\nis out of scope here and should be run once by anyone integrating this\nenvironment into `prime-rl` or the Environments Hub before trusting it in\nproduction. See `env.py`'s own module docstring for the full research trail.\n\nThe baseline runner, `scripts/run_baseline.py`, drives one or more model\nclients through every family's curated instances, tool calls included, and\ngrades every rollout with that family's own deterministic grader:\n\n```\n.venv/bin/python scripts/run_baseline.py                          # dry run, no network calls\n.venv/bin/python scripts/run_baseline.py --limit 3                # fast smoke test\n.venv/bin/python scripts/run_baseline.py --live --confirm-spend   # the real thing, costs money\n```\n\nIt defaults to `--dry-run` (a deterministic stub model, zero network calls, zero\ncost) and refuses to make a single paid request unless both `--live` and\n`--confirm-spend` are passed; the cost estimate is always printed first, and it\nnever shows a made-up dollar figure for a model this repo has no sourced\npricing for. An unpriced model prints as unknown instead; see `baseline/costs.py`\nand the `--pricing-file` flag if you want a real number.\n\nVendor status as of 2026-08-20, checked live from this machine except where\nstated:\n\n- **OpenAI is confirmed working.** `gpt-5.6-luna`, the current frontier tier on\n  the account behind `OPENAI_API_KEY`, returns a correct completion. It is the\n  default model for `--openai-model`.\n- **Anthropic is implemented but not yet usable.** `ANTHROPIC_API_KEY` is not\n  set anywhere on this machine, and there is no default model id: nothing here\n  could verify a Claude model id against a live endpoint without the key.\n  Set `ANTHROPIC_API_KEY` and pass `--anthropic-model` to use it.\n- **Google Gemini is unreachable from this machine.** `GET /v1beta/models`\n  returns a location error, and every current Gemini model id tried 404s. The\n  client is left in place for a location where the API is reachable, but a\n  `--live` run from here will not get results from it.\n- **Prime Intellect Inference is implemented and confirmed reachable, but\n  blocked on account funding, not on code or credentials.** `PRIME_API_KEY`\n  is set and works: `prime inference models` lists the full catalog, and a\n  live smoke call against `moonshotai/kimi-k3` reached Prime's own API and\n  got a real API response. Getting there needed one fix: the shared HTTP\n  helper `_post_json` was sending urllib's default `Python-urllib/x.y`\n  User-Agent, and Cloudflare in front of `api.pinference.ai` returns a bare\n  `HTTP 403 error code: 1010` for that string before the request reaches\n  Prime's API at all. `_post_json` now sends a real User-Agent by default\n  (see `baseline/vendors.py`), confirmed against a live call. The base URL\n  (`https://api.pinference.ai/api/v1`) is confirmed correct by that same\n  call, not just sourced from docs. What is left is money, not code: both\n  the personal wallet and the `SolarSight` team wallet show a $0.00 balance\n  (`prime wallet`), so every request past the Cloudflare check now returns a\n  clean `402 insufficient_funds` from Prime's own API. Add funds at\n  https://app.primeintellect.ai/dashboard/billing (personal) or the team's\n  own billing page, then a `--prime-model` run needs no further changes.\n  Like Anthropic, there is no default model id: pass `--prime-model`.\n\nBecause of this, a `--live` run does not need every vendor to be usable: a\nvendor missing a key (or, for Anthropic and Prime, a model id) is skipped with\na printed reason, and the run continues with whatever vendors it does have,\nselectable with `--vendors openai,anthropic,gemini,prime` (the default, all\nfour). The task spec calls for results from at least two different vendors\nbefore a baseline counts as evidence. A run that ends up with fewer says so\nprominently in its own report, both at the top and in the closing summary,\nrather than reading as if it met that bar.\n\nAPI keys are read from `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` / `GEMINI_API_KEY`\n/ `PRIME_API_KEY` in the environment or from `~/.codylabs/secrets.env`, never\nhardcoded, never logged. Results write incrementally to\n`baseline/results/<run-id>/rollouts.jsonl`, so a crashed or rate-limited run\ncan be resumed by re-running the same command.\n\n## Provenance and licensing posture\n\nThe physics (PVGIS) is a free public API; the incentive rules are public\ngovernment policy. Neither carries third-party intellectual property this\nenvironment needs to protect or license. SolarSight's own engine was used only\nas the verification oracle during development, transcribed test assertions and\nall; nothing in this repository imports or depends on SolarSight's code at\nruntime, per `TASK-V1-ENVIRONMENT.md`'s non-negotiables.\n\nThis repository is licensed under the MIT License (see `LICENSE`). That\ncovers everything shipped here, meaning the code and the public split's\nground truth; `splits/private/`, the held-out set, never ships at all (see\n\"Public/private splits\" above and `tests/test_packaging.py`), so the licence\nquestion does not arise for it.\n\n## What the baseline shows\n\nThe first live baseline ran 2026-08-18, run id `luna-final`: OpenAI\n`gpt-5.6-luna`, reasoning effort medium, all 75 curated public instances,\nzero crashed, zero truncated. Full numbers, per-dimension breakdowns and the\nfailure taxonomy are in `baseline/RESULTS-2026-08-18.md`. The headline:\n\n| Family | Mean | Solved |\n| --- | --- | --- |\n| T1 yield-estimate | 0.999 | 16/16 |\n| T2 orientation-allocate | 0.765 (regraded; 0.922 under the pre-R3 weights) | 9/14 |\n| T3 incentive-eligibility | 0.749 | 8/16 |\n| T4 tou-payback | 0.379 | 0/13 |\n| T5 proposal-audit | 0.600 | 3/16 |\n\n**T1 is a solved control: 16 of 16 solved, mean 0.999, at ceiling, and it\ndoes not discriminate.** The spec's own rule applies here: a family scoring\nnear 100% is too easy and worth nothing, and that should be said plainly\nrather than shipped quietly. T1 should be read as a warm-up or a control, not\nas evidence of difficulty, and it is a candidate for being dropped or made\nsubstantially harder. The evidence that the convention traps are real, not\nfake difficulty, is the reasoning-on-versus-off contrast: the same model,\n`gpt-5.6-luna`, with reasoning disabled falls straight into the\naspect-conversion trap and scores 0.55 on the same task design (an earlier\nsmoke test, `baseline/results/smoke1/`). With reasoning on, medium effort, it\nsolves all 16. So the traps genuinely catch a non-reasoning model; they are\njust not hard for a frontier model that is allowed to reason.\n\nT2's allocation solver is also effectively solved (value_fraction 0.989,\nfeasible 1.000); the family's real signal is all in distinguishes_objectives,\nat 0.643 (and 0 of 5 on the instances that can actually test it: control sites\nand yield-objective instances cannot fail this dimension by construction). The\ngrader has been reweighted to reflect that (`baseline/RESULTS-2026-08-18.md`'s\nR3 section, 2026-08-20, `pyproject.toml` 0.2.0): value_fraction/feasible/\ndistinguishes_objectives moved from 0.6/0.2/0.2 to 0.25/0.10/0.65, derived\nfrom each dimension's measured spread in `luna-final`, not chosen a priori.\nUnder the new weights `luna-final`'s T2 mean is 0.765, not 0.922; the 0.922\nfigure was flattering the family exactly as predicted. T3 identifies which incentive\nschemes apply almost perfectly (0.938 precision and recall) but computes\ntheir value poorly (0.504), which is a different failure than the spec\nexpected and worth noting as such. T5 is the strongest result: it finds\nevery planted error (recall 1.000) but over-flags clean proposals\n(precision 0.517), which is exactly the calibration failure it was built to\nmeasure.\n\nT4's 0.379 is still the family with the weakest oracle behind it: its ground\ntruth depends partly on an hourly generation shape this repo authored, now\nmeasured against PVGIS's real hourly data rather than unvalidated (see the\nlimitations section above): a mean 6.8% divergence on northern-hemisphere\ninstances, 72.2% on southern-hemisphere ones. Its npv_accuracy of\n0.091 has been audited, though (`baseline/RESULTS-2026-08-18.md`'s R2\nsection, 2026-08-20): the tolerance band is not too tight. In all 13 T4\nrollouts the agent's NPV error was larger than the smallest known\nwrong-formula decoy error for that same instance, so the low score reflects\ngenuine multi-step arithmetic failure, not an unfairly tight band. The\nprompt payload's wording was clarified to state its discounting/escalation/\ndegradation conventions explicitly, closing a minor under-specification gap\nthat audit surfaced, but that fix is too small to explain the measured\nscore. Full reasoning for all five families is in\n`baseline/RESULTS-2026-08-18.md`.\n\nThe overall mean across all 75 instances, regraded under the R3 T2 weights,\nis 0.710 (0.739 under the pre-R3 weights). Do not read that number on its own:\nit hides T1's ceiling, and it should not be quoted without the per-family\ntable above.\n\n## Status\n\nA baseline has been run, `luna-final`, 2026-08-18: see \"What the baseline\nshows\" above and `baseline/RESULTS-2026-08-18.md` for the full result. This\nremains preliminary. It is a single vendor, OpenAI's `gpt-5.6-luna`, and the\ntask spec's two-vendor bar is **not met**.\n\nGemini is geo-blocked from this location (`GET /v1beta/models` returns a\nlocation error, and every current Gemini model id tried 404s). Anthropic is\nimplemented but unusable here because `ANTHROPIC_API_KEY` does not exist on\nthis machine. The planned second vendor is Prime Intellect's own inference\nAPI (`PrimeClient` in `baseline/vendors.py`), and this environment has now\nbeen published there (see \"From the Prime Intellect Environments Hub\"\nabove), so the account and key exist. The Prime client itself is confirmed\nworking end to end against the live API as of 2026-08-20: a Cloudflare\nblock caused by urllib's default User-Agent was found and fixed (see\n`baseline/vendors.py`'s vendor-status comment), and a smoke call reached\nPrime's own API and got a real, well-formed API response. That response was\n`402 insufficient_funds`: both the personal wallet and the `SolarSight` team\nwallet behind this account show a $0.00 balance (`prime wallet`). Nothing\nleft to build or debug here, only funds to add, at\nhttps://app.primeintellect.ai/dashboard/billing. Once the wallet has a\nbalance, run:\n\n```\n.venv/bin/python scripts/run_baseline.py --live --confirm-spend --vendors prime --prime-model moonshotai/kimi-k3\n```\n\n`moonshotai/kimi-k3` is this repo's current pick for the second vendor: the\nhighest-priced, most recently released model among Prime's current-generation\nopen-weight families (DeepSeek, Qwen, Kimi, GLM) as of 2026-08-20, chosen\nover OpenAI's or Anthropic's own models available through Prime because the\npoint of a second vendor is a genuinely different model family, not the same\ntwo labs again. It has not yet been exercised past a `402`, so treat it as a\nstarting pick, not a verified one: if it turns out not to support tool\ncalling the way this repo's rollout loop expects, the smoke-test step in\n`scripts/run_baseline.py`'s own instructions (`--families t1 --limit 2`)\nwill show that cheaply before a full 75-instance run is paid for. Until a\nsecond vendor's full run actually completes, this environment's difficulty\nclaims remain a single-vendor data point and do not meet the spec's\ntwo-vendor bar. See \"How to run it\" above for the full per-vendor detail.\n\n## TODO\n\nEverything outstanding, grouped by kind. This section is the single place to\nlook for outstanding work in this repo, per the portfolio convention: no\nseparate `TODO.md`. The detailed spec for all outstanding work, in priority\norder, is `TASK-V2-ROADMAP.md`; read it before picking up any item below.\n\n### Done\n\n1. Prime Intellect account created, `prime login` completed. The account is\n   a Team account (`SolarSight`, slug `solar-sight`) with no personal\n   username set; the team slug is the publishing namespace instead, and\n   nothing here needed a personal username to work.\n\n2. Prime API key created with the Inference permission and stored as\n   `PRIME_API_KEY` in `~/.codylabs/secrets.env`. Confirmed working live:\n   `prime inference models` lists the full catalog.\n\n3. Pushed to the Environments Hub with `prime env push`, public visibility:\n   https://app.primeintellect.ai/dashboard/environments/solar-sight/solar-eval.\n   The `<owner>` placeholder in the README's Hub install instructions above\n   is replaced with the real namespace, `solar-sight`.\n\n### Blocked on the owner\n\n4. Confirm or change the MIT licence. It was a default choice, not a\n   considered decision. Rationale recorded here: the public split going out\n   permissively is consistent with the spec, and the 880-instance private\n   held-out split is the actual moat and never ships.\n\n5. Add funds to the Prime wallet so the second-vendor baseline can actually\n   run. Both the personal wallet and the `SolarSight` team wallet show\n   $0.00 (`prime wallet`), and every call past the Cloudflare/User-Agent fix\n   below now returns a clean `402 insufficient_funds` from Prime's own API.\n\n   https://app.primeintellect.ai/dashboard/billing\n\n### Close the spec's last definition of done item\n\n6. The two-vendor bar is still unmet, but everything except money is now in\n   place. `PrimeClient` reaches Prime's live API correctly as of 2026-08-20:\n   a Cloudflare block caused by urllib's default User-Agent string was found\n   and fixed in `baseline/vendors.py`'s shared `_post_json` helper (commit\n   in this repo's history), confirmed by a live smoke call that got a real\n   `402 insufficient_funds` response instead of a bare `403 error code:\n   1010`. Once item 5 above is done, run:\n\n   ```\n   .venv/bin/python scripts/run_baseline.py --live --confirm-spend --vendors prime --prime-model moonshotai/kimi-k3 --families t1 --limit 2 --run-id prime-smoke\n   ```\n\n   first as a smoke test (inspect the rollouts: did the model attempt the\n   task, call tools, and get graded), then the full run:\n\n   ```\n   .venv/bin/python scripts/run_baseline.py --live --confirm-spend --vendors prime --prime-model moonshotai/kimi-k3 --run-id prime-final\n   ```\n\n   Then update `baseline/RESULTS-2026-08-18.md` and this README with real\n   two-vendor numbers, and remove the preliminary banner caveat. Note Gemini\n   is geo-blocked from this machine: the models list endpoint returns a\n   location error, and every current model id tried 404s. The owner has\n   declined to create an Anthropic key, which is why Prime is the second\n   vendor. `moonshotai/kimi-k3` is this repo's current pick, chosen for\n   being the highest-priced, most recently released model among Prime's\n   current-generation open-weight families as of 2026-08-20 (see the\n   Status section above); it has not been run past a smoke test, so treat\n   the choice as a starting point, not a verified one.\n\n### Product decisions raised by the baseline\n\n7. T1 is at ceiling: 0.999, 16 of 16 solved, by `gpt-5.6-luna` with\n   reasoning on. The spec says a family at ceiling is worth nothing. It\n   currently ships labelled as a solved control. Decide whether to harden it\n   (harder geometries, compounding multi-plane traps, traps that survive\n   reasoning) or drop it. Evidence that the traps are real but easy: the\n   same model with reasoning disabled scored 0.55 and fell into the\n   aspect-conversion trap.\n\n8. DONE (2026-08-20, `TASK-V2-ROADMAP.md` R3). T2 was half at ceiling:\n   `value_fraction` 0.989 and `feasible` 1.000 meant the allocation\n   optimisation was solved, while all the discriminating signal was in\n   `distinguishes_objectives` at 0.643 (0 of 5 on the instances that can\n   actually test it). Reweighted the grader from that measured spread:\n   0.6/0.2/0.2 to 0.25/0.10/0.65\n   (value_fraction/feasible/distinguishes_objectives). `luna-final`'s T2\n   mean under the new weights is 0.765, not 0.922. See\n   `baseline/RESULTS-2026-08-18.md`'s R3 section and\n   `scripts/regrade_t2_reweight.py`, the committed regrade script.\n\n### Technical follow-ups\n\n9. DONE (2026-08-20, `TASK-V2-ROADMAP.md` R2). T4's `npv_accuracy` is 0.091\n   with 0 of 13 solved. Audited whether the NPV tolerance band was simply\n   too tight: it is not. In all 13 rollouts the agent's NPV error exceeded\n   the smallest known wrong-formula decoy error for that instance, so the\n   score reflects genuine arithmetic failure. The prompt payload's wording\n   was clarified to state the discounting/escalation/degradation\n   conventions explicitly, a minor under-specification fix, not a scoring\n   change. See `baseline/RESULTS-2026-08-18.md`'s R2 section. T4's ground\n   truth still rests partly on the newly authored hourly PV shape that has\n   no oracle (see item 12 below), which is a separate, still-open question\n   from the tolerance band.\n\n10. DONE (2026-08-20, `TASK-V2-ROADMAP.md` R5). Added\n    `tests/test_crossvalidation.py`, importing the same test functions and\n    case lists each domain module's own test file already defines, as the\n    single runnable demonstration of the spec's cross-validation\n    non-negotiable.\n\n11. DONE (2026-08-20, `TASK-V2-ROADMAP.md` R6). The T3 Canada instance had\n    real ported eligibility dates but no jurisdiction-specific assertion in\n    SolarSight's TypeScript suite to cross-validate against, unlike AU, US,\n    GB and IE. Fixed upstream: `incentives.test.ts` gained a\n    `ca-federal-greener-homes-grant` worked example\n    (github.com/daviddigital/solarsight PR #147, merged).\n\n12. PHASE 1 DONE (2026-08-20, `TASK-V2-ROADMAP.md` R4), phase 2 open.\n    Measured T2/T4's shared hourly PV shape against PVGIS's real hourly\n    data: northern-hemisphere instances diverge a mean 6.8% of system cost\n    on T4 NPV (inside the 20% loose band); southern-hemisphere instances\n    diverge a mean 72.2% (far outside it). See\n    `baseline/RESULTS-2026-08-18.md`'s R4 section for the full numbers and\n    why a full ground-truth swap was judged disproportionate to what the\n    data shows is wrong (localized to southern hemisphere, not uniform).\n    Swapping the ground truth for the affected instances, or narrowing the\n    model specifically for southern-hemisphere geometries, is still open.\n","encoding":"utf-8","truncated":false,"total_bytes":37383},"status":null}