{"data":{"kind":"file","path":"README.md","version_id":"xvz54u65k8b7fu4d0wr52cw1","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5706,"modified_at":"2026-08-07T06:30:03.617000","content_hash":"664ce079f57f2f2f8b608dbc6fee0491f70b36ac05ca8ff9843040d5401e54c7"},"entries":[],"content":"# sklearn-env\r\n\r\n### Overview\r\n- **Environment ID**: `sklearn-env`\r\n- **Short description**: RL environment for scikit-learn modelling and evaluation, graded by array comparison\r\n- **Tags**: scikit-learn, machine-learning\r\n\r\n### Datasets\r\n- **Primary dataset(s)**: `eltociear/sklearn-tasks-v1`\r\n- **Source links**: [HuggingFace Dataset](https://huggingface.co/datasets/eltociear/sklearn-tasks-v1)\r\n- **Split sizes**: train (25 examples)\r\n\r\n### What this actually tests\r\n\r\nNot array algebra — the numeric siblings already cover that. This tests knowing which\r\nestimator, which preprocessing and above all which **evaluation** is correct. Fitting a scaler\r\nbefore the split, reporting accuracy on a 90/10 imbalanced problem, or letting a pipeline leak\r\nthrough cross-validation are the mistakes this domain punishes, and the tasks are chosen to\r\nexpose them.\r\n\r\n| Category | Tasks | Description |\r\n|----------|-------|-------------|\r\n| preprocessing | 6 | StandardScaler, MinMaxScaler, PCA explained variance, SelectKBest, stratified split shapes |\r\n| classification | 6 | LogisticRegression, RandomForest, DecisionTree importances, SVC pipeline, GradientBoosting, raw predictions |\r\n| validation | 5 | KFold CV, a scaler fitted **inside** each fold, score mean/std, KMeans inertia and cluster sizes |\r\n| metrics | 4 | accuracy vs precision/recall/F1 on an imbalanced set, ROC AUC from `predict_proba`, confusion counts, class balance |\r\n| regression | 4 | R²/MAE/RMSE, coefficients, Ridge, scaled Ridge pipeline |\r\n\r\n### Task\r\n- **Type**: Multi-turn tool use (default `max_turns=5`)\r\n- **Rubric overview**: Binary pass/fail using `numpy.allclose` against the reference array\r\n\r\n### Quickstart\r\n\r\n```bash\r\nuv run vf-eval sklearn-env -p prime -m openai/gpt-5.4-nano -s\r\n```\r\n\r\n### Tools Available\r\n- `execute_code(code: str)` — the three seeded datasets (`X_clf/y_clf`, `X_imb/y_imb`,\r\n  `X_reg/y_reg`) are preloaded, `np` and `sklearn` imported, and `result` persists across turns\r\n- `bash(command: str)` — run shell commands in the sandbox\r\n\r\n### What makes ML gradeable at all\r\n\r\n* **Every dataset is generated from a fixed seed** — no download, no network. The sandbox\r\n  rebuilds them with **exactly the calls the prompt names**, and that reconstruction is verified\r\n  to reproduce the answer-key data element for element, rather than shipping a serialised copy\r\n  that could drift from the generator the task describes.\r\n* **Every estimator with a `random_state` gets one, and the prompt states it.** An unseeded\r\n  RandomForest is not a gradeable answer.\r\n* **References return small arrays of numbers**, rounded where rounding is safe — never a\r\n  fitted model object.\r\n* **`n_jobs=1` everywhere.** Thread scheduling changes floating-point reduction order; an answer\r\n  key that depends on the core count of the machine that built it is not an answer key.\r\n\r\n### Building the dataset\r\n\r\n```bash\r\npython build_tasks.py --verify          # check every task\r\npython build_tasks.py --out train.jsonl # regenerate\r\n```\r\n\r\n`--verify` checks each task for clean execution, **determinism across two fresh runs** (which is\r\nwhat catches an unseeded estimator or a thread-order dependence), finite JSON-safe numbers,\r\nnon-emptiness, and an exact serialisation round-trip. All 25 pass on scikit-learn 1.9.0.\r\n\r\n## Environment arguments\r\n\r\n`load_environment()` deliberately exposes very little: the program's guidance is that an\r\nenvironment should have one correct way to be run, so nothing about the prompts, the parsing or\r\nthe grading is configurable.\r\n\r\n| Argument | Default | Meaning |\r\n|---|---|---|\r\n| `split` | `'train'` | Dataset split to load. |\r\n| `dataset_name` | `'eltociear/sklearn-tasks-v1'` | Hugging Face dataset of tasks. Change only to point at a fork. |\r\n| `max_turns` | `5` | Tool-use turns the model gets before the rollout ends. |\r\n| `**kwargs` | — | Passed through to the underlying `SandboxEnv`. |\r\n\r\n## Reward rubric\r\n\r\n| Reward function | Weight | What it returns |\r\n|---|---|---|\r\n| `correctness` | 1.0 | Binary: 1.0 when the answer matches the reference, else 0.0. |\r\n\r\nThere is no LLM judge and no partial credit. The score is computed on the host in\r\n`post_rollout` and read back by the rubric, so the reward is a deterministic function of the\r\nvalues the model left in `result`. A **harness** failure (dead sandbox, unreadable read-back)\r\nalso scores 0.0 but additionally sets `state[\"scoring_error\"]`, so an eval run can tell a\r\nbroken harness from a wrong answer instead of blaming the model.\r\n\r\n## Dependencies\r\n\r\n`datasets>=4.1.0`, `numpy>=1.26.0`, `scikit-learn>=1.4.0`, `verifiers>=0.1.8`\r\n\r\n## Sample `vf-eval` usage\r\n\r\n```bash\r\nuv run vf-install sklearn-env\r\nuv run vf-eval -s sklearn-env -m gpt-4.1 -n 5 -r 3\r\nuv run vf-tui                      # inspect the outputs/ folder it writes\r\n```\r\n\r\n### Known limitation\r\n\r\nThe Docker **sandbox transport** has not been executed — `docker run`, `pip install`, and the\r\nmodel's code running inside the container — because that needs a Docker runtime and an\r\ninference provider key, neither available where this was authored.\r\n\r\nEverything else is exercised. `environments/test_scoring_path.py` runs this environment's\r\n`post_rollout` and `Rubric` with the sandbox mocked out, asserting that a correct answer scores\r\n1.0, a well-formed wrong answer scores 0.0, and a dead sandbox scores 0.0 *and* sets\r\n`scoring_error` so a harness failure is never mistaken for a bad model. It also checks that\r\nwhat `build_tasks.py` emits is exactly what the scorer expects. The environment mirrors the\r\nstructure of `polars_env` (already accepted into the Environments Program) and imports cleanly\r\nagainst `verifiers` 0.2.1.\r\n","encoding":"utf-8","truncated":false,"total_bytes":5706},"status":null}