{"data":{"kind":"file","path":"README.md","version_id":"kvur7yr7ahhypce3rrgl6sma","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6498,"modified_at":"2026-08-07T06:30:03.627000","content_hash":"262dee385e04fca765f2c9c9e04b0bd3fbae0dac178af6d4a2a90d9ccf2377a5"},"entries":[],"content":"# matplotlib-env\r\n\r\n### Overview\r\n- **Environment ID**: `matplotlib-env`\r\n- **Short description**: RL environment for Matplotlib/Seaborn, graded on values read off the axes — never on pixels\r\n- **Tags**: matplotlib, seaborn, data-visualization\r\n\r\n### Datasets\r\n- **Primary dataset(s)**: `eltociear/matplotlib-tasks-v1`\r\n- **Source links**: [HuggingFace Dataset](https://huggingface.co/datasets/eltociear/matplotlib-tasks-v1)\r\n- **Split sizes**: train (27 examples)\r\n\r\n### Task Categories\r\n\r\n| Category | Tasks | Description |\r\n|----------|-------|-------------|\r\n| line_plots | 6 | ydata/xdata readback, multiple lines, legend texts, linestyle and marker, plotting a derived series |\r\n| axes_config | 6 | explicit limits, title and axis labels, explicit ticks, subplot grids, log scale, limits not filtering data |\r\n| bar_charts | 5 | heights, widths, `barh`, stacked bars via `bottom`, explicit tick labels |\r\n| statistical | 5 | histogram counts and edges, custom bin edges, scatter offsets, boxplot median and whiskers |\r\n| seaborn | 5 | `lineplot` passthrough, `barplot` group means, `estimator=\"sum\"`, `scatterplot` collections, `histplot` heights |\r\n\r\n### ⚠ Why this environment does not compare images\r\n\r\nImage comparison is the obvious way to grade a plot and the wrong one: it fails on font\r\nhinting, DPI, backend and antialiasing long before it tests whether the model plotted the right\r\nthing. Here the model builds the figure and the grade reads values back off the artists.\r\n\r\nThat still leaves a determinism trap, and it was **measured** on matplotlib 3.11.1, not assumed:\r\n\r\n```\r\nln.get_color()  ->  (0.1215…, 0.4666…, 0.7058…)   from the STYLE CYCLE — version-dependent\r\nax.get_xlim()   ->  (-0.2, 4.2) after autoscale   depends on margin defaults\r\nln.get_ydata()  ->  exactly what was passed in    stable\r\nhist counts     ->  computed from the data        stable\r\nax.get_xlim()   ->  after ax.set_xlim(0, 10)      stable — the task set it\r\n```\r\n\r\n**Every task grades only** user-supplied data, computed values, and properties the task itself\r\nsets explicitly. Never a default colour, never an autoscaled limit, never an automatic tick\r\nlocation. The sandbox forces the **Agg** backend: headless, and no interactive-backend\r\ndifferences.\r\n\r\n### Task\r\n- **Type**: Multi-turn tool use (default `max_turns=5`)\r\n- **Rubric overview**: Binary pass/fail on a row comparison — exact on order, arity and type; floats to a narrow tolerance\r\n\r\n### Quickstart\r\n\r\n```bash\r\nuv run vf-eval matplotlib-env -p prime -m openai/gpt-5.4-nano -s\r\n```\r\n\r\n### Tools Available\r\n- `execute_code(code: str)` — `plt`, `np`, `sns` imported and the task's data preloaded as\r\n  ordinary variables; `result` persists across turns\r\n- `bash(command: str)` — run shell commands in the sandbox\r\n\r\nThe model must leave `result` as a **list of lists of plain values**, casting numpy scalars\r\nwith `float()`/`int()`.\r\n\r\n### Grading\r\n\r\nRow order, row count, arity and the string/number distinction are compared **exactly** — a\r\nmodel returning `\"3.0\"` where a float was asked for fails. Floats compare with a narrow relative\r\ntolerance, because several answers here are *computed* (histogram counts, boxplot medians,\r\nseaborn's per-group means) and there is no reason to fail a correct figure over a last-bit\r\ndifference.\r\n\r\nThe comparator has its own unit checks rather than being trusted: wrong order fails, last-bit\r\nnoise passes, a real difference fails, `\"3.0\"` does not pass for `3.0`, wrong arity fails.\r\n\r\nA wrong answer scores `0.0` silently. A broken scorer scores `0.0` and additionally sets\r\n`state[\"scoring_error\"]`.\r\n\r\n### Building the dataset\r\n\r\n```bash\r\npython build_tasks.py --verify          # check every task\r\npython build_tasks.py --out train.jsonl # regenerate\r\n```\r\n\r\n`--verify` checks each task for clean execution, determinism **across two freshly built\r\nfigures**, a non-empty list of JSON-safe primitives — which catches an Artist or numpy scalar\r\nleaking out — no NaN/inf, and an exact serialisation round-trip. All 27 pass on matplotlib\r\n3.11.1 / seaborn 0.13.2.\r\n\r\n## Environment arguments\r\n\r\n`load_environment()` deliberately exposes very little: the program's guidance is that an\r\nenvironment should have one correct way to be run, so nothing about the prompts, the parsing or\r\nthe grading is configurable.\r\n\r\n| Argument | Default | Meaning |\r\n|---|---|---|\r\n| `split` | `'train'` | Dataset split to load. |\r\n| `dataset_name` | `'eltociear/matplotlib-tasks-v1'` | Hugging Face dataset of tasks. Change only to point at a fork. |\r\n| `max_turns` | `5` | Tool-use turns the model gets before the rollout ends. |\r\n| `**kwargs` | — | Passed through to the underlying `SandboxEnv`. |\r\n\r\n## Reward rubric\r\n\r\n| Reward function | Weight | What it returns |\r\n|---|---|---|\r\n| `correctness` | 1.0 | Binary: 1.0 when the answer matches the reference, else 0.0. |\r\n\r\nThere is no LLM judge and no partial credit. The score is computed on the host in\r\n`post_rollout` and read back by the rubric, so the reward is a deterministic function of the\r\nvalues the model left in `result`. A **harness** failure (dead sandbox, unreadable read-back)\r\nalso scores 0.0 but additionally sets `state[\"scoring_error\"]`, so an eval run can tell a\r\nbroken harness from a wrong answer instead of blaming the model.\r\n\r\n## Dependencies\r\n\r\n`datasets>=4.1.0`, `matplotlib>=3.8.0`, `seaborn>=0.13.0`, `verifiers>=0.1.8`\r\n\r\n## Sample `vf-eval` usage\r\n\r\n```bash\r\nuv run vf-install matplotlib-env\r\nuv run vf-eval -s matplotlib-env -m gpt-4.1 -n 5 -r 3\r\nuv run vf-tui                      # inspect the outputs/ folder it writes\r\n```\r\n\r\n### Known limitation\r\n\r\nThe Docker **sandbox transport** has not been executed — `docker run`, `pip install`, and the\r\nmodel's code running inside the container — because that needs a Docker runtime and an\r\ninference provider key, neither available where this was authored.\r\n\r\nEverything else is exercised. `environments/test_scoring_path.py` runs this environment's\r\n`post_rollout` and `Rubric` with the sandbox mocked out, asserting that a correct answer scores\r\n1.0, a well-formed wrong answer scores 0.0, and a dead sandbox scores 0.0 *and* sets\r\n`scoring_error` so a harness failure is never mistaken for a bad model. It also checks that\r\nwhat `build_tasks.py` emits is exactly what the scorer expects. The environment mirrors the\r\nstructure of `polars_env` (already accepted into the Environments Program) and imports cleanly\r\nagainst `verifiers` 0.2.1.\r\n","encoding":"utf-8","truncated":false,"total_bytes":6498},"status":null}