{"data":{"kind":"file","path":"README.md","version_id":"vs2cwokpfqzr023b7nnpfblh","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6831,"modified_at":"2026-08-07T06:30:03.610000","content_hash":"5fe2c744badac1c2e11ce9a7642063fcd1c5140b4025c008c593e34b41dce85c"},"entries":[],"content":"# django-orm-env\r\n\r\n### Overview\r\n- **Environment ID**: `django-orm-env`\r\n- **Short description**: RL environment for Django ORM queries, graded by exact row comparison\r\n- **Tags**: django, orm, sql\r\n\r\n### Datasets\r\n- **Primary dataset(s)**: `eltociear/django-tasks-v1`\r\n- **Source links**: [HuggingFace Dataset](https://huggingface.co/datasets/eltociear/django-tasks-v1)\r\n- **Split sizes**: train (29 examples)\r\n\r\n### Task Categories\r\n\r\n| Category | Tasks | Description |\r\n|----------|-------|-------------|\r\n| filtering | 9 | `__gte`/`__lt`/`__in`/`__range`/`__contains`, booleans, `Q` objects, `exclude` |\r\n| aggregation | 8 | `count`, `Count`/`Sum`/`Max`/`Avg` annotations, HAVING-style filters, distinct counts |\r\n| relations | 6 | forward and reverse FK traversal, filtering across two relations, `distinct()` after a join |\r\n| expressions | 6 | `F` expressions, ordering by a related field, top-N slicing, `aggregate` |\r\n\r\n### Why this environment exists alongside the numeric ones\r\n\r\nThe siblings (pandas, numpy/scipy, PyTorch, JAX) are all numeric. An agent that can do tensor\r\nalgebra may still be unable to write a correct ORM query, and ORM mistakes are their own skill:\r\na missing `distinct()` after a join, an aggregate that fans out across a relation, a filter\r\napplied on the wrong side of a FK.\r\n\r\n### Task\r\n- **Type**: Multi-turn tool use (default `max_turns=5`)\r\n- **Rubric overview**: Binary pass/fail on an exact row-by-row comparison (floats to a tolerance)\r\n\r\n### Quickstart\r\n\r\n```bash\r\nuv run vf-eval django-orm-env -p prime -m openai/gpt-5.4-nano -s\r\n```\r\n\r\n### Environment Arguments\r\n\r\n| Arg | Type | Default | Description |\r\n|-----|------|---------|-------------|\r\n| `split` | str | `\"train\"` | Dataset split to use |\r\n| `dataset_name` | str | `\"eltociear/django-tasks-v1\"` | HuggingFace dataset name |\r\n| `max_turns` | int | `5` | Maximum interaction turns per task |\r\n\r\n### Tools Available\r\n- `execute_code(code: str)` — runs with Django configured, the database populated, and\r\n  `Publisher`, `Author`, `Book`, `Q`, `F`, `Count`, `Sum`, `Avg`, `Max` in scope\r\n- `bash(command: str)` — run shell commands in the sandbox\r\n\r\nThe model must leave `result` as a **list of tuples of plain values**, not a QuerySet.\r\n\r\n### Why ORM work is gradeable at all\r\n\r\n* **One in-memory-style SQLite database from a hard-coded fixture** — no network, no migrations\r\n  on disk, no clock or randomness in the data.\r\n* **References return lists of primitives, never QuerySets**, so answers compare by value rather\r\n  than by object identity or lazy evaluation.\r\n* **Every task orders explicitly.** SQLite promises nothing about row order without an\r\n  `ORDER BY`; grading an unspecified order would be grading luck.\r\n\r\nRow order, row count and strings are compared **exactly**. Only numbers carry a tolerance,\r\nbecause Decimal averages return floats whose last digit can differ by representation and failing\r\na correct query over that would be grading arithmetic noise. The comparator has its own unit\r\nchecks — including one asserting that a correct set of rows in the **wrong order fails**.\r\n\r\n### Building the dataset\r\n\r\n```bash\r\npython build_tasks.py --verify          # check every task\r\npython build_tasks.py --out train.jsonl # regenerate\r\n```\r\n\r\n`--verify` checks each task for clean execution, determinism **across two freshly rebuilt\r\ndatabases**, a non-empty list of JSON-safe primitives, that it is not the trivial \"return every\r\nrow\" answer, and an exact serialisation round-trip. All 29 pass on Django 6.0.8.\r\n\r\nIt earned its keep here: it caught a task asking for *\"every year with at least 2 books\"*, which\r\nreturned **nothing** — every book in the fixture has a distinct year. Replaced with the same\r\n`GROUP BY … HAVING` shape over the publisher's country, which has real duplicates and exercises\r\na join.\r\n\r\n### ⚠ Why the distribution is named `django-orm-env`, not `django-env`\r\n\r\n`django-env` **already exists on PyPI** (v0.2.1, an unrelated virtualenv manager). The Hub's own\r\nsuggested install line keeps PyPI as the primary index:\r\n\r\n```\r\nuv pip install <name> --extra-index-url https://hub.primeintellect.ai/eltociear/simple/\r\n```\r\n\r\nso a colliding distribution name silently installs a stranger's package instead of this one.\r\nThat is exactly what happened on the first push, and it was caught by installing the published\r\nwheel rather than trusting the source tree — see `environments/test_hub_wheels.py`.\r\n\r\n## Environment arguments\r\n\r\n`load_environment()` deliberately exposes very little: the program's guidance is that an\r\nenvironment should have one correct way to be run, so nothing about the prompts, the parsing or\r\nthe grading is configurable.\r\n\r\n| Argument | Default | Meaning |\r\n|---|---|---|\r\n| `split` | `'train'` | Dataset split to load. |\r\n| `dataset_name` | `'eltociear/django-tasks-v1'` | Hugging Face dataset of tasks. Change only to point at a fork. |\r\n| `max_turns` | `5` | Tool-use turns the model gets before the rollout ends. |\r\n| `**kwargs` | — | Passed through to the underlying `SandboxEnv`. |\r\n\r\n## Reward rubric\r\n\r\n| Reward function | Weight | What it returns |\r\n|---|---|---|\r\n| `correctness` | 1.0 | Binary: 1.0 when the answer matches the reference, else 0.0. |\r\n\r\nThere is no LLM judge and no partial credit. The score is computed on the host in\r\n`post_rollout` and read back by the rubric, so the reward is a deterministic function of the\r\nvalues the model left in `result`. A **harness** failure (dead sandbox, unreadable read-back)\r\nalso scores 0.0 but additionally sets `state[\"scoring_error\"]`, so an eval run can tell a\r\nbroken harness from a wrong answer instead of blaming the model.\r\n\r\n## Dependencies\r\n\r\n`datasets>=4.1.0`, `django>=5.0`, `verifiers>=0.1.8`\r\n\r\n## Sample `vf-eval` usage\r\n\r\n```bash\r\nuv run vf-install django-orm-env\r\nuv run vf-eval -s django-orm-env -m gpt-4.1 -n 5 -r 3\r\nuv run vf-tui                      # inspect the outputs/ folder it writes\r\n```\r\n\r\n### Known limitation\r\n\r\nThe Docker **sandbox transport** has not been executed — `docker run`, `pip install`, and the\r\nmodel's code running inside the container — because that needs a Docker runtime and an\r\ninference provider key, neither available where this was authored.\r\n\r\nEverything else is exercised. `environments/test_scoring_path.py` runs this environment's\r\n`post_rollout` and `Rubric` with the sandbox mocked out, asserting that a correct answer scores\r\n1.0, a well-formed wrong answer scores 0.0, and a dead sandbox scores 0.0 *and* sets\r\n`scoring_error` so a harness failure is never mistaken for a bad model. It also checks that\r\nwhat `build_tasks.py` emits is exactly what the scorer expects. The environment mirrors the\r\nstructure of `polars_env` (already accepted into the Environments Program) and imports cleanly\r\nagainst `verifiers` 0.2.1.\r\n","encoding":"utf-8","truncated":false,"total_bytes":6831},"status":null}