{"data":{"kind":"file","path":"README.md","version_id":"nuhtb9eul1knw2y85eiqhut4","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":8677,"modified_at":"2026-07-25T14:37:19","content_hash":"843845e314e9dc3929a157a2446d74d12afbfed6cc502337e3a3f377f2976128"},"entries":[],"content":"# fastapi-competency\n\nWrite a FastAPI app from a written spec. The grader imports the module, drives it with\n`TestClient`, and scores how much of the specified HTTP behaviour actually holds.\n\nNothing is graded by reading the source. Every point comes from a real request\nand a real response, so a model that describes the right approach without implementing\nit scores zero, and a model that reaches the right behaviour by an unusual route still\nscores full marks.\n\n## What is in the suite\n\n34 tasks, 113 checks. Each task names the routes, the status codes, the JSON shapes and\n(where it matters) the headers and the OpenAPI document that the app has to produce.\n\nCoverage follows the FastAPI docs, not a random walk over the API surface:\n\n| Area | Examples |\n| --- | --- |\n| Routing and parameters | path operation ordering, numeric and string validations, repeated query params, `convert_underscores` on headers, cookies |\n| Bodies | Pydantic models, `Body(embed=True)`, several body params plus a singular value, nested models with `HttpUrl` and `set`, `PATCH` with `exclude_unset` |\n| Responses | `response_model` filtering, `response_model_exclude_unset`, `response_model_by_alias`, 204 with an empty body, changing the status code through the injected `Response` |\n| Errors | `HTTPException` with custom headers, a handler for a custom exception class, overriding the `RequestValidationError` and `HTTPException` handlers |\n| Dependencies | shared query dependencies, per-request caching and `use_cache=False`, dependencies on the decorator, `yield` teardown ordering |\n| App structure | `APIRouter` prefixes and tags, mounted sub-applications, a custom `APIRoute` class, HTTP middleware, background tasks, lifespan |\n| Schema and streaming | path operation metadata in `/openapi.json`, an extended OpenAPI document, OAuth2 password bearer, server-sent events with `fastapi.sse` |\n\nA fair number of checks read `/openapi.json` instead of a normal endpoint. Those are\nthe checks that are hardest to satisfy by accident, because the generated schema only\ncomes out right when the parameter was declared the way FastAPI expects.\n\n## Install and run\n\n```bash\nuv run vf-install fastapi-competency\nuv run vf-eval fastapi-competency -m gpt-5-mini -n 5 -r 3\n```\n\nAgainst a specific provider:\n\n```bash\nuv run vf-eval fastapi-competency -p anthropic -m claude-opus-4-5-20251101 -n 34 -r 2 -s\n```\n\nOnly the hard tier:\n\n```bash\nuv run vf-eval fastapi-competency -a '{\"difficulty\": \"hard\"}' -m gpt-5-mini\n```\n\n### Dependencies\n\n`verifiers`, `datasets`, `fastapi>=0.135`, `httpx`, `python-multipart`. The FastAPI floor\nmatters. One task uses `fastapi.sse`, which arrived in 0.135, and the form and OAuth2\ntasks need `python-multipart`. `load_environment` checks for all of it up front and\nraises there, instead of failing later inside a rollout.\n\nNo API keys, no dataset download, no network access, no container.\n\n### Arguments\n\n| Argument | Default | Meaning |\n| --- | --- | --- |\n| `timeout_seconds` | 90 | wall clock budget for executing one submission |\n| `difficulty` | `None` | keep only `easy`, `medium` or `hard` tasks |\n\n## Reward rubric\n\n| Function | Weight | Value |\n| --- | --- | --- |\n| `functional_correctness` | 1.0 | fraction of the task's checks that passed |\n| `module_imported` | 0.0 | 1.0 if the submission imported and exposed `app` |\n| `checks_passed` | 0.0 | raw count of passing checks |\n\nReward is the fraction of checks, so it lands on a real scale instead of pass or fail.\nTasks carry 2 to 6 checks each. A check passes only when every expectation attached to\nit holds, and a submission that fails to import scores 0.\n\nGrading happens once per rollout in a subprocess and is cached on the rollout state. The\ntwo zero weighted functions are metrics for reading result files, they do not move the\nreward.\n\n## Running untrusted code\n\nThe grader executes model output. Read this before pointing it at anything you do not\ntrust.\n\nEach submission is written to a fresh temporary directory and run as a separate process\nin its own session. The child caps its own CPU time and address space, the parent\nenforces a wall clock timeout and kills the whole process group when it expires, and the\nchild environment is stripped of every variable whose name looks like a credential, so\nthe submission cannot read the API key the evaluation is running on.\n\nThat is process isolation, not a sandbox. There is no filesystem or network jail. If you\nare evaluating models you have no reason to trust, run the whole evaluation inside a\ncontainer or a throwaway VM.\n\n### Keeping the answer key away from the policy\n\nWorth knowing if you train on this, because a policy under gradient pressure will find\nthese before a person does.\n\nThe checks for a task reach the child over stdin, so there is no answer key sitting on\ndisk beside the submission. The module cache is then purged of this package and an\nimport hook refuses to load it, so a submission cannot pull the task table out of\n`fastapi_competency.tasks` either. Both routes were tested: a submission that reads\n`checks.json` and one that imports the task table both score 0.0 on a task they would\notherwise have partly solved.\n\nWhat is left is a plain file read of the installed package inside site-packages. Closing\nthat needs a filesystem sandbox, which is out of scope for an environment that is\notherwise dependency free. The realistic answer for adversarial training runs is to grade\ninside a container.\n\n## How the tasks were checked\n\nEvery expectation in this suite was written against the FastAPI docs and then confirmed\nagainst a reference implementation. All 34 reference solutions score exactly 1.0 on all\n113 checks, which is what rules out expectations that no correct app could satisfy.\n\nFour negative controls run the other direction: an empty submission, a module with no\n`app`, a module with a syntax error, and a catch-all app answering every route with an\nempty object. Mean scores across the suite are 0.000, 0.000, 0.000 and 0.010. So the\nchecks are not passing on accident, and the rubric spans the range.\n\n## Observed scores\n\nAll runs are 34 examples, 2 rollouts each, FastAPI 0.140 on Python 3.12.\n\n| Model | Reward | easy (5) | medium (12) | hard (17) |\n| --- | --- | --- | --- | --- |\n| claude-opus-4-5 | 0.971 | 1.000 | 1.000 | 0.941 |\n| claude-haiku-4-5 | 0.875 | 0.667 | 0.938 | 0.892 |\n| gpt-4.1-nano | 0.840 | 1.000 | 0.868 | 0.772 |\n\nPer rollout the reward lands on 0.0, 0.333, 0.5, 0.667, 0.75 and 1.0, so partial credit\nis doing real work. Across the 34 tasks the stronger model wins 6, loses 1 and ties 27.\n\nThe aggregate gap is narrow, and the reason is worth stating plainly. Both of these are\nfrontier models and FastAPI is extremely well represented in their training data, so most\nof the suite is at ceiling for both. Where the weaker model loses, it loses on genuine\nframework mistakes: putting a Pydantic `Field` on a path parameter instead of `Path`, and\nannotating a repeated query parameter as `list[str] = []` without `Query()`, which quietly\nturns it into a request body. The stronger model drops its only task on SSE, where it\ninvents import paths for `EventSourceResponse` that do not exist.\n\nAdding a third and much cheaper model makes the picture readable. Overall reward falls\n0.971, 0.875, 0.840 in the order you would want, and on the hard tier it falls 0.941,\n0.892, 0.772 across 17 tasks.\n\n**How much to trust the third decimal, measured rather than guessed.** The nano row was\nrun twice at these settings and came back 0.794 and 0.840. Same model, same suite, a\nspread of about 0.05. Two rollouts per task is enough to rank three models that are far\napart and nowhere near enough to separate two that are close. If you need to tell apart\nmodels within about 0.05 of each other, raise `-r` well above 2 and expect to spend for\nit. Every number in that table is a single run.\n\nIt also kills a claim an earlier draft of this file made. That draft said the easy tier\nseparated models better than the hard tier, on the reasoning that the framework-specific\nidioms live there. The third model breaks it: `gpt-4.1-nano` scores 1.000 on the easy\ntier, above `claude-haiku-4-5` at 0.667. The easy tier is 5 tasks, so 10 rollouts, and a\nsingle bad rollout moves it 0.1. Read the hard tier if you want signal and treat the easy\ncolumn as too small to rank anything on.\n\nThe honest summary is that the suite discriminates on the hard tier, sits near ceiling on\nthe rest for any current frontier model, and should spread further on smaller open models,\nwhich is the setting it is meant for. That last part is still a prediction and it is the\none claim here nobody has measured.\n","encoding":"utf-8","truncated":false,"total_bytes":8677},"status":null}