{"data":{"kind":"file","path":"README.md","version_id":"iybd4xjb8ugxvb95pr8eem6d","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6991,"modified_at":"2026-08-07T06:30:03.642000","content_hash":"361003a483de6379c314961b8d27971ea985e4299619c5a5ca6700e64c366c39"},"entries":[],"content":"# csapp-env\r\n\r\nRL environment for **machine-level computer systems**, on the topics of *Computer Systems: A\r\nProgrammer's Perspective* (Bryant & O'Hallaron): two's complement, integer overflow, IEEE-754\r\nfloating point, bit manipulation, memory layout and caches.\r\n\r\n- **Tasks:** 40 across 5 categories (integers 12, floats 10, bits 6, memory 6, cache 6)\r\n- **Dataset:** [`csapp-tasks-v1`](https://huggingface.co/datasets/eltociear/csapp-tasks-v1)\r\n- **Reward:** binary, from an exact row comparison. No LLM judge, no network, no clock.\r\n- **Dependencies in the sandbox:** none. Every task is answered with `struct` and `math`.\r\n\r\n```bash\r\nprime env install eltociear/csapp-env\r\n```\r\n\r\n## Original problems, not the book's exercises\r\n\r\nNothing here is copied from CS:APP. Every task is written fresh on the topics the book teaches.\r\n\r\nThat is a copyright decision, and it is also the stronger evaluation choice. The book's\r\nexercises and their worked solutions are all over the public web, so a model can **recall**\r\nthem instead of computing. Original problems with executed answer keys cannot be recalled —\r\nthe only way to the answer is to work it out.\r\n\r\n## What it measures\r\n\r\nThe places where the obvious answer is wrong:\r\n\r\n- `-1` read as an unsigned 32-bit int is `4294967295`, and `(5u - 9u) > 0` is **true**;\r\n- `x >> 3` sign-extends for a signed int and does not for an unsigned one — while Python's\r\n  `>>` is *always* arithmetic and its ints are unbounded, so C's fixed width has to be\r\n  emulated deliberately rather than assumed;\r\n- `INT_MIN` has no positive counterpart: negating it wraps back to itself;\r\n- float32 addition is **not associative** — with `a = 1e8, b = -1e8, c = 1.0`, `(a+b)+c` is\r\n  `1.0` and `a+(b+c)` is `0.0`;\r\n- `2**24 + 1` is a whole number that float32 **cannot hold**; it rounds to `2**24`;\r\n- `sizeof {char; int; char}` is **12**, not 6 — and reordering to `{int; char; char}` makes it\r\n  8;\r\n- on a 2-set direct-mapped cache, reading an `int a[4][8]` column-major misses **32** times\r\n  against row-major's **8**.\r\n\r\n## Every answer key is executed, never typed\r\n\r\nThe expected output of each task is produced by running a reference implementation, so a key\r\ncan never drift from the instruction beside it.\r\n\r\nThat rule caught four defects in this environment's own first draft, all of which `--verify`\r\nor a hand check surfaced before publishing:\r\n\r\n| Task | What was wrong |\r\n|---|---|\r\n| non-associativity | the prompt said \"show that float32 addition is not associative\" and **both groupings came out `0.0`** — the chosen values did not demonstrate it. Now asserted in the builder: the task fails to build if the two sides ever agree. |\r\n| row vs column major | with a cache large enough to hold the array, both orders tie at the compulsory-miss count (8 and 8). The comparison was empty. The cache is now deliberately smaller than the working set, and the builder asserts the two counts differ. |\r\n| cache hit/miss trace | the access sequence produced **five misses and no hits**, so a model answering `0` five times scored full marks. The sequence now contains a genuine hit and a genuine conflict miss, and that is asserted. |\r\n| struct layout | the offsets were **four typed constants**, the one thing this project says it never does. Replaced with an actual alignment-and-padding algorithm. |\r\n\r\n## Grading\r\n\r\nExact, with no tolerance anywhere — the opposite decision from the numeric sibling\r\nenvironments (`pytorch-env`, `jax-env`, `numpy-scipy-env`), which use `allclose` because\r\neigensolvers and BLAS differ in their last bits. Nothing here is measured, sampled, timed or\r\nseeded: every answer is an integer, a bit pattern, a boolean, or a float read back from its own\r\nbit pattern. There is nothing for a tolerance to absorb, and on a question *about exact bit\r\npatterns* a tolerance would only widen the set of wrong answers that pass.\r\n\r\nTwo comparisons a naive `actual == expected` would get wrong, and which the comparator handles:\r\n\r\n- **`1` must not pass for `True`** — many tasks return a boolean beside an integer, and\r\n  Python's `True == 1` would hide a confused answer.\r\n- **`\"0x2a\"` must not pass for `42`** — the hex-string convention is part of what is asked.\r\n\r\n## Verify it yourself\r\n\r\n```bash\r\npython environments/csapp_env/build_tasks.py --verify   # 40/40\r\n```\r\n\r\n`--verify` re-checks every task independently: that it runs, that it is **deterministic across\r\ntwo fresh executions**, that it returns a non-empty list of JSON-safe primitives, that any hex\r\nstring it emits really is lowercase, and that it survives the serialisation round-trip exactly.\r\n\r\n## Environment arguments\r\n\r\n`load_environment()` deliberately exposes very little: the program's guidance is that an\r\nenvironment should have one correct way to be run, so nothing about the prompts, the parsing or\r\nthe grading is configurable.\r\n\r\n| Argument | Default | Meaning |\r\n|---|---|---|\r\n| `split` | `'train'` | Dataset split to load. |\r\n| `dataset_name` | `'eltociear/csapp-tasks-v1'` | Hugging Face dataset of tasks. Change only to point at a fork. |\r\n| `max_turns` | `5` | Tool-use turns the model gets before the rollout ends. |\r\n| `**kwargs` | — | Passed through to the underlying `SandboxEnv`. |\r\n\r\n## Reward rubric\r\n\r\n| Reward function | Weight | What it returns |\r\n|---|---|---|\r\n| `correctness` | 1.0 | Binary: 1.0 when the answer matches the reference, else 0.0. |\r\n\r\nThere is no LLM judge and no partial credit. The score is computed on the host in\r\n`post_rollout` and read back by the rubric, so the reward is a deterministic function of the\r\nvalues the model left in `result`. A **harness** failure (dead sandbox, unreadable read-back)\r\nalso scores 0.0 but additionally sets `state[\"scoring_error\"]`, so an eval run can tell a\r\nbroken harness from a wrong answer instead of blaming the model.\r\n\r\n## Dependencies\r\n\r\n`datasets>=4.1.0`, `verifiers>=0.1.8`\r\n\r\n## Sample `vf-eval` usage\r\n\r\n```bash\r\nuv run vf-install csapp-env\r\nuv run vf-eval -s csapp-env -m gpt-4.1 -n 5 -r 3\r\nuv run vf-tui                      # inspect the outputs/ folder it writes\r\n```\r\n\r\n## Known limitation\r\n\r\nThe Docker **sandbox transport** has not been executed — `docker run`, `pip install`, and the\r\nmodel's code running inside the container — because that needs a Docker runtime and an\r\ninference provider key, neither available where this was authored.\r\n\r\nEverything else is exercised. `environments/test_scoring_path.py` runs this environment's\r\n`post_rollout` and `Rubric` with the sandbox mocked out, asserting that a correct answer scores\r\n1.0, a well-formed wrong answer scores 0.0, and a dead sandbox scores 0.0 *and* sets\r\n`scoring_error` so a harness failure is never mistaken for a bad model. It also checks that\r\nwhat `build_tasks.py` emits is exactly what the scorer expects. The environment mirrors the\r\nstructure of `polars_env` (already accepted into the Environments Program) and imports cleanly\r\nagainst `verifiers` 0.2.1.\r\n","encoding":"utf-8","truncated":false,"total_bytes":6991},"status":null}