{"data":{"kind":"file","path":"README.md","version_id":"qixhkgvcjnfhut04n1oc43xl","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6864,"modified_at":"2026-08-07T06:30:03.637000","content_hash":"3cc70cbfbd5db726456345ccc6cd42bc4d85a35ae53aa21d258a9a8b8f7cd204"},"entries":[],"content":"# fastapi-env\r\n\r\nRL environment for **FastAPI**: routing, request validation, response models and dependency\r\ninjection — graded by comparing real HTTP responses, exactly.\r\n\r\n- **Tasks:** 25 across 5 categories (routing 5, validation 8, responses 5, dependencies 4,\r\n  schema 3)\r\n- **Dataset:** [`eltociear/fastapi-tasks-v1`](https://huggingface.co/datasets/eltociear/fastapi-tasks-v1)\r\n- **Reward:** binary, from an exact row comparison. No LLM judge, no network, no clock.\r\n\r\n```bash\r\nprime env install eltociear/fastapi-env\r\n```\r\n\r\n## Why this environment exists\r\n\r\nFastAPI's behaviour is **implicit**. Nothing in the source of a handler tells you that:\r\n\r\n- a missing required query parameter is **422**, not 400;\r\n- an unparseable typed path parameter (`/items/abc` against `item_id: int`) is also 422;\r\n- `POST` to a `GET`-only route is **405**, and FastAPI answers it without touching your code;\r\n- a `response_model` **silently removes** fields your handler returned;\r\n- a `Depends` that raises `HTTPException` short-circuits **before** the handler body runs;\r\n- `status_code=201` on the decorator changes the success code but not the error codes.\r\n\r\nThose are exactly the things a model gets wrong while writing code that *looks* right. They\r\nare only visible in the response, which is what this environment grades.\r\n\r\n## What makes HTTP gradeable\r\n\r\n`fastapi.testclient.TestClient` runs the application **in process** — no server, no port, no\r\nnetwork, no clock. This environment exists because that was initially missed: FastAPI was\r\nwritten off as \"needs a running server, so it cannot be verified locally\". That was wrong, and\r\nthe mistake cost real buildable work until it was caught.\r\n\r\nBeyond that:\r\n\r\n- every app is constructed **inside its own task**; nothing is shared and nothing persists, so\r\n  no ordering effect can change an answer;\r\n- `result` is a list of rows of JSON primitives read **off the response** — a status code, a\r\n  header value, a JSON scalar — never a `Response` object.\r\n\r\n## Grading\r\n\r\nComparison is **exact**, and that is a deliberate difference from the numeric sibling\r\nenvironments (`pytorch-env`, `jax-env`, `numpy-scipy-env`) which use `allclose`. Nothing graded\r\nhere is a computed float: status codes, header strings and JSON scalars are exact quantities,\r\nso a tolerance would buy nothing and would only widen the set of wrong answers that pass.\r\n\r\nThe comparator has its own unit checks (`test_compare.py`, 17/17) rather than being trusted.\r\nThe two cases that matter are the ones a naive `actual == expected` gets **wrong**:\r\n\r\n- **`1` must not pass for `True`.** `{\"pong\": true}` and `{\"pong\": 1}` are different JSON\r\n  responses, but Python's `True == 1` is true and would hide it.\r\n- **`\"200\"` must not pass for `200`.** A model that reads the status code out as a string has\r\n  not read the response correctly, and it looks identical in a printout.\r\n\r\nRow order, row count and row width are all significant too.\r\n\r\n## The trap this environment was built around\r\n\r\nA Pydantic body model must be defined at **module level**. The task builder uses\r\n`from __future__ import annotations`, which turns every annotation into a string; FastAPI then\r\nresolves `p: Plant` against the module globals. A model defined **inside** the function is not\r\nresolvable that way, and FastAPI does not raise — it **silently reclassifies the request body\r\nas a query parameter**. Measured, same code both ways:\r\n\r\n```\r\nwithout `from __future__ import annotations`  ->  200  {\"name\": \"fern\", \"height_cm\": 30}\r\nwith    `from __future__ import annotations`  ->  422  {\"loc\": [\"query\", \"p\"], \"type\": \"missing\"}\r\n```\r\n\r\nThe first draft of this environment defined the model inside the task and would have shipped\r\nthat 422 as the *expected answer* for a task whose prompt describes a working echo endpoint.\r\n`--verify` is what caught it. The system prompt now tells the model the same rule, because its\r\ncode is `exec`'d and is subject to the identical resolution problem.\r\n\r\n## Verify it yourself\r\n\r\n```bash\r\npython environments/fastapi_env/build_tasks.py --verify   # 25/25, rebuilds every answer key\r\npython environments/fastapi_env/test_compare.py           # 17/17 comparator checks\r\n```\r\n\r\n`--verify` independently re-checks every task: that it runs, that it is **deterministic across\r\ntwo fresh applications**, that it returns a non-empty list of JSON-safe primitives, and that it\r\nsurvives the serialisation round-trip exactly.\r\n\r\n## Environment arguments\r\n\r\n`load_environment()` deliberately exposes very little: the program's guidance is that an\r\nenvironment should have one correct way to be run, so nothing about the prompts, the parsing or\r\nthe grading is configurable.\r\n\r\n| Argument | Default | Meaning |\r\n|---|---|---|\r\n| `split` | `'train'` | Dataset split to load. |\r\n| `dataset_name` | `'eltociear/fastapi-tasks-v1'` | Hugging Face dataset of tasks. Change only to point at a fork. |\r\n| `max_turns` | `5` | Tool-use turns the model gets before the rollout ends. |\r\n| `**kwargs` | — | Passed through to the underlying `SandboxEnv`. |\r\n\r\n## Reward rubric\r\n\r\n| Reward function | Weight | What it returns |\r\n|---|---|---|\r\n| `correctness` | 1.0 | Binary: 1.0 when the answer matches the reference, else 0.0. |\r\n\r\nThere is no LLM judge and no partial credit. The score is computed on the host in\r\n`post_rollout` and read back by the rubric, so the reward is a deterministic function of the\r\nvalues the model left in `result`. A **harness** failure (dead sandbox, unreadable read-back)\r\nalso scores 0.0 but additionally sets `state[\"scoring_error\"]`, so an eval run can tell a\r\nbroken harness from a wrong answer instead of blaming the model.\r\n\r\n## Dependencies\r\n\r\n`datasets>=4.1.0`, `fastapi>=0.115.0`, `httpx>=0.27.0`, `verifiers>=0.1.8`\r\n\r\n## Sample `vf-eval` usage\r\n\r\n```bash\r\nuv run vf-install fastapi-env\r\nuv run vf-eval -s fastapi-env -m gpt-4.1 -n 5 -r 3\r\nuv run vf-tui                      # inspect the outputs/ folder it writes\r\n```\r\n\r\n## Known limitation\r\n\r\nThe Docker **sandbox transport** has not been executed — `docker run`, `pip install`, and the\r\nmodel's code running inside the container — because that needs a Docker runtime and an\r\ninference provider key, neither available where this was authored.\r\n\r\nEverything else is exercised. `environments/test_scoring_path.py` runs this environment's\r\n`post_rollout` and `Rubric` with the sandbox mocked out, asserting that a correct answer scores\r\n1.0, a well-formed wrong answer scores 0.0, and a dead sandbox scores 0.0 *and* sets\r\n`scoring_error` so a harness failure is never mistaken for a bad model. It also checks that\r\nwhat `build_tasks.py` emits is exactly what the scorer expects. The environment mirrors the\r\nstructure of `polars_env` (already accepted into the Environments Program) and imports cleanly\r\nagainst `verifiers` 0.2.1.\r\n","encoding":"utf-8","truncated":false,"total_bytes":6864},"status":null}