{"data":{"kind":"file","path":"README.md","version_id":"clevukucnb5deukuyewyv1vg","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":7784,"modified_at":"2026-08-03T15:46:06.175000","content_hash":"3e456aa6f8f9e3fbecc9ea671ea3434ff0864b971f685db6d21f2bf9e9eb7912"},"entries":[],"content":"# compg-bench\n\n`compg-bench` is a native `verifiers.v1` coding-agent environment for\ncomputational geometry primitives. Version 0.5.0 contains 21 tasks: seven\nintersection primitives, each presented in C++20, Rust 2021, and Python 3.\n\nThe current primitives are:\n\n- N-dimensional aligned box/aligned box;\n- 2D line/line, line/ray, line/segment;\n- 2D ray/ray, ray/segment; and\n- 2D segment/segment.\n\nThe query semantics and cases are adapted from Geometric Tools Library headers\nand unit tests (Boost Software License 1.0).\n\n## Install and inspect\n\nFrom the repository root, install the local environment and resolve its native\nv1 configuration:\n\n```bash\nprime --plain env install compg-bench --path environments\nuv run --project environments/compg_bench \\\n  eval compg-bench --dry-run --no-push\n```\n\nThe dry run loads `CompgBenchTaskset` and its bundled `CompgBenchHarness`,\nvalidates all typed configuration, and writes the resolved configuration\nwithout creating a sandbox or calling a model.\n\nThe default runtime is Prime, pinned to the supported `us` region and paced to\nat most eight sandbox creations per minute. It uses the public image\n`docker.io/library/rust:1.88.0-bookworm`, whose toolchain includes G++ 12, Rust\n1.88, and Python 3.11. A task requests one CPU, 2 GB of memory, and 5 GB of\ndisk. The framework owns startup, timeout handling, and shielded cleanup.\n\nAn equivalent self-built image is provided in `images/toolchain/Dockerfile`:\n\n```bash\ndocker build -t compg-toolchain:0.2.0 images/toolchain\n./scripts/validate_local.sh compg-toolchain:0.2.0\n```\n\n## Validate without a model\n\nThe repository-level validator compiles and runs the known-good solution for\nevery task/language pair and checks that generated harnesses are current:\n\n```bash\n./scripts/validate_local.sh\n```\n\nThe v1 lifecycle smoke test additionally creates real Prime runtimes, invokes\n`Task.setup`, installs a known-good solution, invokes `Task.finalize` and\nscoring, and stops each runtime in a `finally` block. Start with one primitive:\n\n```bash\nuv run --project environments/compg_bench \\\n  python scripts/smoke_prime_sandboxes.py \\\n  --task-id line2_line2 cpp rust python\n```\n\nOmit the task and language filters to exercise all 21 rows. `--docker-image`\nselects another pullable image.\n\n## Run an evaluation\n\nThe reproducible acceptance configuration lives at\n`configs/eval/compg-bench.toml`. Validate it, then run it:\n\n```bash\nuv run --project environments/compg_bench \\\n  eval @ configs/eval/compg-bench.toml --dry-run\nuv run --project environments/compg_bench \\\n  eval @ configs/eval/compg-bench.toml\n```\n\nThe config evaluates every row once with at most eight concurrent rollouts and\ndoes not upload the resulting traces. Its whole-rollout retries are restricted\nto sandbox provisioning failures. Runtime calls already perform their own\ntransient retries; functional or task-definition failures are not retried.\n\nFor a fast development run, select a primitive and language on the command\nline:\n\n```bash\nuv run --project environments/compg_bench eval compg-bench \\\n  --taskset.task-ids line2_line2 \\\n  --taskset.languages python \\\n  -n 1 -r 1 -c 1 \\\n  --sampling.max-tokens 4096 \\\n  --sampling.temperature 0.2 \\\n  --no-push\n```\n\nValid task IDs are `aligned_box_aligned_box`, `line2_line2`, `line2_ray2`,\n`line2_segment2`, `ray2_ray2`, `ray2_segment2`, and\n`segment2_segment2`. Valid languages are `cpp`, `rust`, and `python`. Stable\ndataset order is task order followed by C++, Rust, and Python.\n\n## What is scored\n\n| Signal | Kind | Meaning |\n| --- | --- | --- |\n| `task_passed` | reward, weight 1.0 | Compilation succeeds and every hidden functional test passes |\n| `compile_success` | metric | The candidate and hidden harness compiled |\n| `performance_score` | metric | Reserved for a later microbenchmark component; currently zero |\n\nAligned boxes are closed solids, so face, edge, and point contact count as\nintersection. The 2D queries distinguish the applicable none, point, segment,\nray, and coincident cases. Directions need not be normalized, and the tests\npreserve the reference algorithms' exact zero checks for parallelism.\n\nPublic starter files and visible tests are uploaded in `Task.setup`. The hidden\nsuite remains on the verifier host until the agent finishes and is written only\nin `Task.finalize`. C++ and Rust submissions are compiled there; Python is\nsyntax-checked before its hidden suite runs. Grading output and diagnostic\nsignals are copied to the trace before scoring.\n\n## Shared test corpus\n\nTest inputs and expected results are authored once per primitive and visibility\nin `assets/<task_id>/cases/public.json` and `hidden.json`. Language-specific\nharnesses are generated from those shared cases and idiomatic templates. After\nchanging a corpus or template, regenerate and check it with:\n\n```bash\npython3 environments/compg_bench/tools/generate_tests.py\npython3 environments/compg_bench/tools/generate_tests.py --check\n```\n\nFor a standalone Hub source checkout whose root is this environment directory,\nuse `python3 tools/generate_tests.py`. Generated files are dependency-free in\nthe runtime sandbox.\n\n## v1 architecture\n\n```text\ntyped TaskData row\n    -> framework creates Runtime\n    -> CompgBenchTask.setup uploads public files\n    -> built-in BashHarness runs the coding-agent loop\n    -> CompgBenchTask.finalize uploads and runs hidden tests\n    -> @reward and @metric methods score the Trace\n    -> framework stops the Runtime\n```\n\n`compg_bench/taskset.py` owns immutable task data, setup/finalize hooks, and\nscoring. `compg_bench/harness.py` is a small configuration specialization of\nthe built-in v1 Bash harness, used to bundle the paced Prime runtime and retain\nthe original Bash-only tool surface. No task code imports or directly manages\n`prime_sandboxes`.\n\n`assets/<task_id>/cases` is the canonical case corpus. Generated public files\nunder `assets/<task_id>/public/<language>` are visible to the agent; generated\nhidden files stay on the verifier host. `tests/fixtures` contains known-good\nimplementations used only by validator tooling.\n\n## Add another primitive\n\n1. Define equivalent, idiomatic public APIs for all three languages.\n2. Add one language-neutral public and hidden JSON corpus under\n   `assets/<task_id>/cases`.\n3. Reuse or add the three harness templates and register the renderer in\n   `tools/generate_tests.py`.\n4. Generate the harnesses; do not hand-edit generated case blocks.\n5. Add known-good fixtures and run `./scripts/validate_local.sh`.\n6. Add the task prompt to `TASK_TEMPLATES` in `compg_bench/taskset.py`.\n7. Run a filtered v1 lifecycle smoke test, then a one-rollout model evaluation.\n\nThe taskset automatically produces the three language rows. Grading metadata\nonly changes when a new primitive needs different filenames or commands.\n\n## Publish\n\nAfter all validation gates pass, publish a public release to Environment Hub:\n\n```bash\nprime --plain env push compg-bench \\\n  --path environments \\\n  --visibility PUBLIC\n```\n\nThe Hub environment and runtime image are separate registry objects; this\nrelease deliberately uses a public Docker Hub base image.\n\n## Planned performance reward\n\n`performance_score` is an intentional zero-valued seam. A future implementation\nshould run only after correctness passes, build optimized native artifacts,\nwarm up, sample repeatedly, and compare a robust statistic against a reference\nin the same runtime. Inputs should be fixed, seeded, checksummed, and hidden\nuntil finalization. Scores should be normalized within each language and capped\nso raw Python time is never compared directly with native-code time.\n\nA suitable bounded reward is\n`correctness + weight * correctness * normalized_performance`: an incorrect or\nflaky implementation remains at zero regardless of speed.\n","encoding":"utf-8","truncated":false,"total_bytes":7784},"status":null}