{"data":{"kind":"file","path":"README.md","version_id":"urjgp1r62gqcdhl3bcdau55b","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6778,"modified_at":"2026-07-28T08:56:20","content_hash":"a3240a13d0ed1788f6913e910bd8b115bf7ba9ce64a00611c29454cd1ee53da2"},"entries":[],"content":"# go-debug-bench\n\nMulti-file Go debugging environments for RL training and agent evaluation, built on the\n[verifiers](https://pypi.org/project/verifiers/) spec.\n\nEach task is a small, plausible Go module containing exactly one defect that is gated\nbehind knowledge of Go runtime or language semantics rather than surface-level pattern\nmatching. The agent gets a sandbox, a shell, and the repository. Grading writes a hidden\ntest suite into the repository and runs it.\n\n## Why these are hard for models and not for the engineers who know\n\nCoding agents are strong at recognising defects that look like defects. They are weak at\ndefects where every individual file is idiomatic and the fault lives in the relationship\nbetween files. Every task here is built to that shape.\n\n## Design rules\n\nEach rule below exists because breaking it produces a task that measures nothing.\n\n**Hidden tests never ship to the agent.** They are written into the repository at grade\ntime. If the assertions were visible, the fix could be read off them.\n\n**The public test suite passes on the unfixed repository.** A green suite is not evidence\nof correctness. This removes the shortcut of running the tests to locate the fault, and\nit mirrors the situation the task is modelling — a bug that shipped because the tests\ndid not cover it.\n\n**Hidden tests assert on observable behaviour through the public API.** Any correct fix\npasses. No specific implementation is required, so the reward cannot be gamed by\nmatching a known diff.\n\n**Test files are checksummed.** The digest of every `_test.go` file is recorded before\nthe rollout and re-checked after. Editing a test scores zero, closing the \"make the\ntests pass\" degenerate strategy.\n\n**`go vet` is clean on the unfixed repository.** No static-analysis signal points at the\ndefect.\n\n## Reward\n\n| Function | Weight | Meaning |\n|---|---|---|\n| `hidden_tests_pass` | 1.0 | Hidden suite green **and** no test file edited |\n| `compiles` | 0.1 | Repository still builds — partial credit, prevents reward collapse |\n| `tests_untampered` | 0.0 | Metric only, for diagnosing degenerate strategies |\n\n## Tasks\n\nThree, all Go, all rated hard, all two files.\n\n| Task | Knowledge gate | Why the public suite misses it |\n|---|---|---|\n| `eventbus-aliasing` | `append` into a zero-length re-slice writes through the caller's backing array | It never filters anyone out, so every write lands on the index it came from |\n| `worker-pool-leak` | an unbuffered send blocks forever once the only receiver stops reading | It only calls `Collect(0)`, which drains the channel and lets every sender finish |\n| `config-omitempty` | `omitempty` cannot distinguish an explicitly-set zero from an absent field | Every case it round trips has `Enabled: true` and non-zero retries — the exact values `omitempty` preserves |\n\n### `eventbus-aliasing` (hard, 2 files)\n\nAn in-process publish/subscribe router. Disabled subscribers are supposed to stay\nregistered and resume delivery when re-enabled. They do not: after any publish that\nfilters somebody out, the subscriber registry is silently corrupted.\n\nThe knowledge gate is that `append` into a zero-length re-slice writes through the\ncaller's backing array. `filter.go` uses the documented Go idiom for allocation-free\nfiltering — correct when the caller does not reuse the input. `bus.go` assumes its\nregistry survives a call. Neither file is wrong on its own.\n\nWhat makes it resist a model:\n\n- `go vet` is clean\n- the public suite passes, because it never filters anyone out, so every write lands on\n  the index it came from and the corruption is invisible\n- the symptom points at `SetEnabled` or `Subscribe`, not at the filtering helper that\n  actually causes it\n\nReference fix is a one-line allocation change, but any fix that stops `selectTargets`\nwriting through the caller's array passes.\n\n### `worker-pool-leak` (hard, 2 files)\n\nA bounded worker pool. `Collect(limit)` runs queued jobs and returns the first\n`limit` results; callers routinely stop early. The contract is that no workers\nsurvive the call. A service using it grows in memory and exhausts its thread\nlimit under load.\n\nThe knowledge gate is that a send on an unbuffered channel blocks forever once\nthe only receiver has stopped reading. Every goroutine is spawned correctly and\nthe `WaitGroup` is used correctly; the defect is created by the caller's `break`,\nwhich strands every worker still trying to send and means the closer goroutine\nnever runs either.\n\nThe symptom — memory and thread growth in production — points at pool sizing or\nthe jobs, not at the collection loop.\n\n### `config-omitempty` (hard, 2 files)\n\nService configuration with overlays. Operators disable a service during an\nincident and it is written back through `Save`. Absent fields are filled from\ndefaults on load so older documents keep working. Services disabled during an\noutage come back running after the next reload, and a service set to zero\nretries retries three times.\n\nThe knowledge gate is that `omitempty` erases an explicitly-set `false`, after\nwhich the loader cannot distinguish it from absent and the default overwrites the\noperator's decision. Both halves are individually correct: `omitempty` is the\nright choice for a genuinely optional field, and `Region` uses it correctly.\n`go vet` does not analyse struct tags.\n\n## Usage\n\n```bash\nuv pip install go-debug-bench\n```\n\n```python\nimport verifiers as vf\n\nenv = vf.load_environment(\"go-debug-bench\")\n\n# or a single task\nenv = vf.load_environment(\"go-debug-bench\", task_ids=[\"eventbus-aliasing\"])\n```\n\nRequires a Go toolchain in the sandbox image; defaults to `golang:1.24-bookworm`.\n\n## Adding a task\n\n```\ngo_debug_bench/tasks/<task-id>/\n├── task.json      # id, prompt, difficulty, reference fix, why it's hard\n├── repo/          # the Go module the agent sees, including passing public tests\n└── hidden/        # *_test.go files copied in at grade time\n```\n\nBefore adding a task, run the verifier. It checks all four properties across\nevery task and runs the grading matrix over four strategies:\n\n```bash\npython verify_tasks.py\n```\n\n```\n=== worker-pool-leak  (package pool, hard) ===\n    ok   go vet clean\n    ok   public suite passes\n    strategy                    build  tamper  hidden  reward\n    unfixed                      True   False   False     0.1\n    reference fix                True   False    True     1.1\n    cheat: delete a test         True    True   False     0.1\n    broken: does not compile    False   False   False     0.0\n```\n\nA task that fails any check measures something other than debugging skill. The\nPython suite (`pytest`) covers metadata and packing; `verify_tasks.py` needs a Go\ntoolchain and covers the properties that actually matter.\n\n## License\n\nMIT\n","encoding":"utf-8","truncated":false,"total_bytes":6778},"status":null}