{"data":{"kind":"file","path":"README.md","version_id":"jg10agqgmelj4nwnrb9edj62","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":15349,"modified_at":"2026-07-07T04:50:51.946000","content_hash":"7cfb2fcd6ca069f5e664b40a798c1a48c49bfa867fbb29955b3d4a8b36a216ac"},"entries":[],"content":"# nodal-pricing-lmp\n\nLocational marginal pricing (LMP) as a dual-space reasoning task — the\nprice-formation rung of an energy-systems vertical built with the same\nvalidation discipline as its siblings\n([economic-dispatch](https://github.com/JWilksBooth/economic-dispatch),\n[dcopf-grid-verifiers](https://github.com/JWilksBooth/dcopf-grid-verifiers),\n[multiperiod-dispatch](https://github.com/JWilksBooth/multiperiod-dispatch),\n[n1-contingency-dispatch](https://github.com/JWilksBooth/n1-contingency-dispatch)),\nand the parallel robotics vertical built the same way\n([robot-cycle-time](https://github.com/JWilksBooth/robot-cycle-time),\n[bimanual-chore-scheduling](https://github.com/JWilksBooth/bimanual-chore-scheduling)).\nThe vertical's primal environments ask *what should run*; this one asks\n*what is it worth, and where*. Given a congested lossless DC network (buses,\n**directed** lines with reactances and MW flow limits, generators with offer\nprices and capacity, nodal loads), the model must produce the economic\nreadout of the optimal dispatch: nodal LMPs, the energy/congestion\ndecomposition against a pinned reference bus, the binding-line set, and\n**signed** line shadow prices. The model is given instance data only — never\nthe solved dispatch, which would leak the binding set.\n\nThe naive answer this environment defeats is the two-bus intuition that a\nnode's price is bounded by the offer band. In a meshed network that is simply\nfalse: on a measured **44% of instances some bus is priced outside the offer\nband** — below the cheapest offer or above the priciest — because counterflow\non a congested line pays a bus to *withdraw*. A bus priced **below the cheapest\ngenerator's offer** is not a bug; it is what real traders get paid to\nunderstand, and it is unfakeable by band-clipping heuristics.\n\n## Task\n\nGiven the network and offers, output one flat JSON object:\n\n```\n{\"lmp_usd_per_mwh\": [...], \"congestion_usd_per_mwh\": [...],\n \"binding_lines\": [...], \"mu_usd_per_mwh\": [...]}\n```\n\nall prices in $/MWh to 2 decimals. Rewards: format (0.10), consistency\n(0.15 — the four KKT identities), binding_set (0.15 — Jaccard, false-positive\npenalized), lmp_accuracy (0.45 — `exp(−k·mean|ΔLMP|/scale)`), and mu_accuracy\n(0.15). The three graded components are hard-gated on the consistency gate:\nan answer whose internal KKT identities do not hold scores 0 on all price\ncredit no matter how close its numbers look.\n\n## The three-legged ground truth\n\nPrices are validated three ways, and the independence claim is stated exactly\nand no more:\n\n- **Primary (B-θ angle space):** a time-invariant DC-OPF LP in bus angles;\n  LMPs read off the nodal-balance equality duals, signed line shadow prices\n  off the interleaved flow-limit inequality duals (scipy/HiGHS, sign mapping\n  empirically pinned on the hand-computed 3-bus example).\n- **Cross-check (PTDF injection space):** an independently *formulated* LP in\n  the shift-factor (injection) space, built from the spec text alone by an\n  implementer who never read the primary. It reconstructs `LMP_i = λ − Σ_k\n  μ_k·PTDF_ki` and must agree per element.\n- **Certificate (convention-free):** a finite-difference value-function probe.\n  It perturbs each bus load by ±0.5 MW and checks the symmetric difference\n  quotient of the primal objective against the claimed LMP — validating every\n  price against the *definition* of marginal cost using primal objective\n  values only, with no dual convention to inherit a sign bug from.\n\n**Honest independence framing.** The two LP legs share the same scipy/HiGHS\nsolver core (exactly as the dcopf-grid-verifiers sibling shares its solver\nacross its two modeling paths), so a dual-extraction quirk could in principle\nhit both identically — the finite-difference certificate is the leg that\ncannot, because it never touches a dual. This environment therefore claims\n*two independently formulated LPs solved by the same engine, cross-checked by\na solver-agnostic finite-difference certificate* — **never** \"three\nindependent implementations.\" (The pandapower cross-check the dcopf sibling\nuses is dropped here — pandapower does not expose nodal duals cleanly — which\nis what makes the dependency story scipy-only; see\n[CALIBRATION.md](CALIBRATION.md).)\n\n## Ground-truth metrics (N = 200, seeds 0–199)\n\n| Metric | Result |\n|---|---|\n| B-θ vs PTDF leg LMP disagreements | **0 / 200** |\n| B-θ vs PTDF leg μ disagreements | **0 / 200** |\n| Worst leg deltas | \\|ΔLMP\\| 1.3e-12, \\|Δμ\\| 5.9e-12 (tolerance 1e-2) |\n| Finite-difference certificate failures | **0 / 200** |\n| Congestion-rent three-way identity failures | **0 / 200** |\n| Merchandising-surplus / cap violations (\\|LMP\\| > \\$2,000) | 0 / 200 |\n| Congested instances | 152 / 200 (76%) |\n| Two-plus binding lines | 55 / 200 (28%) |\n| Out-of-band (some LMP outside the offer band) | 89 / 200 (44%) |\n| Raw-draw accept rate (survivor-bias indicator) | 52% |\n\nThe congestion-rent identity is checked three ways on every instance —\n`Σ_k μ_k·flow_k` from each leg's duals, and the merchandising surplus\n`Σ_i LMP_i·d_i − Σ_g LMP_bus(g)·p_g` from the primary — all within\n`0.01·total_load` \\$/h. Leg deltas ride HiGHS vertex duals that are exact to\n~1e-12 on integer-quantized data, so the 1e-2 agreement tolerances carry ~10\norders of headroom.\n\n## Signed shadow prices and the energy/congestion decomposition\n\n`mu_usd_per_mwh[k]` is line k's shadow price, **signed by the line's stated\nfrom→to orientation**: `mu[k] = +14.0` means the limit binds in the from→to\ndirection and relaxing it by 1 MW would cut total system cost by \\$14/h;\ncongestion in the to→from direction makes `mu[k]` negative; every non-binding\nline reports exactly 0.\n\nPrices decompose against reference bus 0 and nothing else: `energy := lmp[0]`\nand `congestion[i] := lmp[i] − lmp[0]`, so `congestion[0]` is 0.0 by\ndefinition — a decomposition computed against any other reference bus is a\nwrong answer **here** even though the LMPs themselves are reference-independent.\nThe network is lossless DC, so the marginal-loss component that real ISO\nthree-part LMPs carry is identically zero and omitted.\n\n*Worked example.* On the pinned 3-bus fixture (equal-reactance triangle,\n\\$20 gen at bus 0, \\$50 gen at bus 2, loads [0, 30, 90]) line 2 (0→2, limit\n50 MW) carries exactly +50 and binds. The duals are `LMP = [20, 35, 50]`,\n`mu = [0, 0, +45]`: energy is \\$20 everywhere, bus 2's \\$30 congestion premium\nis the +45 shadow price shifted through the PTDF, and bus 1 sits at \\$35 —\nhalfway, because the triangle splits its withdrawal. Every price is inside the\n[20, 50] offer band *on this instance* — the point of the environment is that\nin a mesh they generically are not.\n\n## The counterflow / out-of-band phenomenon\n\nThe hard, unfakeable case is a bus whose LMP lands **outside** the offer band.\nWhen a line is congested, injecting at a bus that *relieves* that line (a\nnegative-PTDF, counterflow bus) is worth more than any generator's energy, so\nits LMP is pushed **below the cheapest offer** — sometimes negative. The\ntwo-bus practitioner intuition (a real, documented misconception) says a\nnode's price is clipped to `[min offer, max offer]`; a heuristic that clips to\nthe band scores near-exactly on in-band instances and **collapses** on\nout-of-band ones, because the clipped price vector no longer lies in the span\nof the binding lines' PTDF rows and the reconstruction identity (F4) closes\nthe consistency gate. Out-of-band lands **44%** of instances, overwhelmingly\non the **above-priciest** side (~92%) with a low-side / negative-LMP tail\n(~8–12%) — the asymmetry follows the tiered offer stack and is published in\nthe sweep table, not hidden.\n\n## Reward ladder\n\nMedian / mean total reward per baseline, graded through the real reward stack\non the shipped config (N = 60 calibration sweep; full table in\n[CALIBRATION.md](CALIBRATION.md)):\n\n| Baseline | Median | Mean | What it is |\n|---|---|---|---|\n| format-only | 0.10 | — | parseable, no valid identities |\n| self-consistent spam | 0.25 | — | format + consistency floor, zero instance knowledge |\n| all-zeros constant | 0.25 (cong) / 0.55 (uncong) | 0.32 | the zero-knowledge answer |\n| uniform-λ (Tier 1) | **0.250** | 0.41 | network-blind merit-order price |\n| two-bus, binding-set given (Tier 2) | 1.00 | **0.66** | band-clip applied by someone who does the rest of the math right |\n| optimal | 1.00 | 1.00 | the true duals |\n| *zonal-split (descriptive, ungated)* | — | — | *merit + congestion redispatch, priced per local unit — what an engineer does for one binding line; fails on meshed/multi-marginal* |\n\nThe two-bus baseline is deliberately **bimodal** — ~1.0 on in-band instances,\n~0.21 on out-of-band ones (the designed kill) — so its *median* sits at the\nin-band mode (1.0) and hides the separation, while its *mean* (~0.66) shows\nthe real ~0.34 gap to optimal. The **ship gate is measured on the mean**, the\ncorrect instrument for a bimodal baseline (see CALIBRATION.md, DR-C): mean\nsep(uniform-λ → two-bus) = 0.25 and mean sep(two-bus → optimal) = 0.34, both\ncomfortably above the 0.08 floor. The uniform-λ Tier-1 median lands exactly on\nthe 0.25 self-consistent-spam floor: it is fully KKT-consistent (it cannot be\ngate-killed) but carries no price level on the congested 76%, so it is\n*difficulty-killed*, not gate-killed.\n\n**Per-class breakdown** (shipped config): the separation lives on the\nout-of-band and two-plus slices — there the two-bus baseline reads 0.21 (gate\nclosed) against optimal's 1.0 — while on the uncongested class Tier-1 is simply\n*correct* and scores 1.0. This is where the mean's separation comes from and\nwhy the per-class table is published alongside it.\n\n## Anti-reward-hacking (24 regression-tested attack gates)\n\nSingle shared parse gate: format-invalid answers cascade to zero on all five\nrewards (invariant asserted across the whole attack corpus). The redundant\nfields are deliberate: their internal KKT consistency is the cheap-to-verify\nsurface — an answer that launders wrong prices through a self-consistent story\nmust reproduce the exact span structure of the true binding constraints, which\nis as hard as being right. Gates cover: the uniform-λ and two-bus baselines\n(honest partial credit, strictly separated); NaN/Infinity literals,\n>4300-digit integers, bracket bombs (all score 0 without crashing); arity\nattacks killed by fixed lengths; all-lines-binding farming (double wall on two\npayloads); slack-line μ (material → reconstruction wall, sub-tolerance →\nJaccard wall); three sign-flip probes attributed to the identity that kills\neach; reference-bus reframing caught by the `congestion[0]` tripwire;\nepsilon-μ hedging; a series/parallel-line misattribution channel (truth prices,\nwrong line blamed — priced to ~0.25 by the binding-set gate, and kept out of\nshipped instances by the generation-side alternate-support reject); and the\n3-bus hand example pinned on both legs to catch any unit slip or sign\nregression across scipy/HiGHS versions.\n\n## Usage\n\n```bash\npip install -e .                                  # scipy-only: no [validate] extra\nN_INSTANCES=200 python tests/test_validation.py   # two-leg cross-check + FD cert + 24 gates\npython calibration/measure.py --n 200             # difficulty sweep + ship gate\n\n# give frontier models room to reason; -r 1 matches published baselines\nvf-eval nodal-pricing-lmp -p anthropic -m <model> -n 50 -r 1 --max-tokens 16000\n```\n\nLower token budgets truncate reasoning mid-derivation and score as format\nfailures — the dual-space arithmetic is long. The cross-check needs **only\nscipy** — there is no validation-only extra to install, which is the cheapest\ndependency story in the portfolio.\n\n```python\nimport verifiers as vf\nenv = vf.load_environment(\"nodal-pricing-lmp\", num_examples=300)\n# disjoint train/eval: load_environment(1000) + load_environment(200, seed_offset=1000)\n```\n\n## Instance generation\n\nRandom connected meshed networks (spanning tree + chords, 4–10 buses), 3–5\ngenerators drawn from a tiered merit stack with ≥\\$1.00/MWh pairwise offer\nseparation (kills cost-tie degeneracy), loads and limits integer MW. Every\ndraw is solved at generation and put through the full degeneracy pipeline —\nfinite-difference kink detection, weak-μ and μ-rank certificates, and an\nalternate-support span reject — before acceptance; the raw-draw accept rate\n(~52%) is dominated by plain primal-infeasible draws, not the degeneracy\ngates. Class mix on the shipped config: **76% congested, 28% two-plus binding\nlines, 44% out-of-band**. Fully deterministic per seed. The congestion rate is\ndeliberately oversampled relative to a real N-1-planned grid — a realistic\nsample would be mostly trivial uncongested instances and would not exercise\nthe skill.\n\n## Baseline\n\nOpus 4.8, 30 examples, `--max-tokens 16000` (reproduce with the `vf-eval`\nline above):\n\n| Metric | Value |\n|---|---|\n| Mean reward (all 30) | 0.267 |\n| Truncated at 16k tokens | **15 / 30** (mean output 10.1k tokens) |\n| Mean reward on the 15 **completed** rollouts | **0.535** |\n| — exact (1.0) | 6 / 15 |\n| — right structure, wrong binding set (0.21) | 6 / 15 |\n| — consistent but wrong (0.25) | 3 / 15 |\n\nRead honestly: this environment is **hard**, and it discriminates. On the\nrollouts Opus finishes, it lands across a real 0.21→1.0 spread — it parses and\nstays KKT-consistent (format + consistency credit) but on 40% of completed\ninstances misidentifies the binding constraint set, so the binding-gated price\ncredit correctly stays shut. That is the measured skill: identifying which\nlines bind is the hard part, and even a frontier model gets it wrong often.\nThe dominant failure, though, is **truncation** — half the rollouts exhaust\n16k tokens because Opus solves the DC power flow by hand (iterating bus\nangles) instead of reasoning about the dual directly; a token-efficient\nLMP-theory approach fits comfortably. Higher token budgets would lift the\ncompletion rate; the number above is at the published 16k. Opus does not clear\n0.9 on the multi-binding subset, so the environment is not frontier-saturated.\n\n**Calibration status.** A frontier probe (Opus 4.8, above) was run. It shows\nthe **binding-set-correctness gate is the dominant discriminator**, not the\nprice-decay curve: when the binding set is right the prices are essentially\nright too (credit ~1.0), and when it is wrong the price gate is shut — so the\nreward decay constants `K_LMP` (= ln 2) and `K_MU` (= 5.0) are barely\nexercised and stay at their analytically-pinned values (right-structure / ±\\$5\nanswers land ~0.5; the all-zeros and epsilon-μ hedge rows). A denser sweep of\n`K` would only matter for the correct-binding-imperfect-price regime, which\nthe frontier rarely reaches; documented here rather than over-tuned.\n\n## Roadmap\n\nLMP is the price-formation layer future storage and trading environments\nconsume: a battery arbitrage or a virtual-bid environment needs a verified\nnodal price series to grade against, and this is that series. Next rung:\nunit-commitment prices (non-convex pricing / uplift), tracked at\n[n1-contingency-dispatch](https://github.com/JWilksBooth/n1-contingency-dispatch)'s\nsuccessor — the binding-set identification here is the prerequisite skill.\n","encoding":"utf-8","truncated":false,"total_bytes":15349},"status":null}