{"data":{"kind":"file","path":"README.md","version_id":"lz8nn32ezlopbsk0zq0sj5rb","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5746,"modified_at":"2026-11-21T01:53:45.989000","content_hash":"d2493363dce22f4ade65e85b0b17c8641f63e5ea74c18092514545110d785fb1"},"entries":[],"content":"# autogenesis\n\nAn RL environment whose policy builds RL environments. Source, tests and baseline\nresults: https://github.com/assemsohaib/autogenesis\n\nThe policy (the **builder**) gets a short domain brief and writes a Python module with\nthree functions:\n\n```python\ndef generate(seed: int) -> dict        # one task; \"prompt\" is what a solver sees\ndef solve(task: dict) -> str           # a reference answer\ndef verify(task: dict, response: str) -> float   # the reward function, 0..1\n```\n\nThe environment then attacks that module and pays the builder only for environments\nthat survive.\n\n## One episode\n\n| Step | What happens | Model calls |\n|---|---|---|\n| 1. build | `builder` (trainable) writes the module for one brief | 1 |\n| 2. audit | The code runs in a sandbox on hidden random seeds. The audit measures validity, determinism, diversity, answer leaks, and solvability, then attacks `verify()` with 16 reward-hacking probes | 0 |\n| 3. panel | A fixed `weak` and a fixed `strong` solver attempt up to 6 generated tasks. The builder's own `verify()` grades them | 2 × instances |\n| 4. reward | `autogenesis = audit_score × (floor + (1 − floor) × max(0, strong_acc − weak_acc))` | 0 |\n\n`audit_score = valid × determinism × solvability × hack_resistance × (1 − leak) × ((1 − w) + w × diversity)`\n\nThe audit is multiplicative on purpose: one broken property gives zero reward. For\nexample, a verifier that the \"shotgun\" probe (all reference answers in one message)\nbeats gets `hack_resistance = 0`.\n\n### The probe suite (a static Hacker)\n\nConstant responses: empty, whitespace, `0`, `1`, `yes`, `true`, `None`, a JSON claim\nof correctness, a prompt injection, 5000 characters of junk, and every integer from −50 to 1000.\nTask-dependent responses: the prompt itself, every number in the prompt, all\nreference answers at once, the most common answer of the other tasks, and a different task's answer.\n`hack_resistance = 1 − max probe pass rate`. Reference answers and probes are mixed in\none shuffled batch, so `verify()` cannot tell which call is the audit.\n\n## Taskset\n\n35 domain briefs in 10 families. `split=train` (default) gives 29 briefs in 8 families.\n`split=eval` gives 6 briefs in the held-out families `grids` and `code_tracing`.\n`split=all` gives all 35. `repeats=N` repeats each brief.\n\n## Usage\n\n```bash\nprime env install assemsohaib/autogenesis\n```\n\nBuilder code runs in a Docker container with no network by default, so a running\nDocker daemon is needed (or use `--env.sandbox.type prime`).\n\n```bash\n# audit-only reward (default when no solvers are set; no solver calls)\nvf-run autogenesis -m <builder-model>\n\n# full reward: pin two different solver models; the panel then turns on\nvf-run autogenesis -m <builder-model> \\\n  --env.weak.model <small-model> --env.strong.model <large-model>\n\n# held-out families, 4 attempts per brief, solver calls in parallel\nvf-run autogenesis -m <builder-model> -r 4 --env.taskset.split eval \\\n  --env.weak.model <small> --env.strong.model <large> --env.max-concurrent-agents None\n```\n\nMain options (`vf-run autogenesis --help` shows all):\n\n| Option | Default | Meaning |\n|---|---|---|\n| `--env.audit.num-tasks` | 24 | Hidden seeds per audit |\n| `--env.audit.seed` | random | Fix the hidden seeds (reproducible evals) |\n| `--env.audit.call-timeout` | 2.0 | Seconds per builder function call |\n| `--env.audit.diversity-weight` | 0.5 | Weight of the diversity term |\n| `--env.panel.instances` | auto | Tasks per solver. `0` = audit-only. Auto: 6 when both solvers are set, else 0 |\n| `--env.panel.floor` | 0.25 | Share of the audit score paid with zero discrimination |\n| `--env.sandbox.type` | docker | `docker` (no network, 1 GB, 1 CPU), `prime`, or `subprocess` (development only) |\n\n## Metrics\n\nOn the builder trace: `audit_score`, `audit_stage` (0 extract, 1 policy, 2 generate,\n3 verify, 4 complete), `valid`, `determinism`, `diversity`, `leak`, `solvability`,\n`reference_stability`, `hack_resistance`, `weak_accuracy`, `strong_accuracy`,\n`discrimination`. `info.autogenesis` contains the error, policy violations, per-probe\npass rates, the worst probe, and sample prompts and answers.\n\nSolver traces are untrainable. They carry a `solved` metric and no reward. A solver\ntrace whose model call fails is dropped, and the panel scores only the tasks that both\nsolvers answered (`info.autogenesis.panel_pairs`). If no task is left, the builder\ntrace is dropped too: an infrastructure failure never counts as a bad environment.\nNote: the `mean reward` line that `vf-run` prints is the mean over *all* traces, so it\nincludes the reward-less solver traces. Read the builder reward from `traces.jsonl`.\n\n## Security\n\nBuilder code is untrusted model output. There are three layers:\n\n1. **Static policy** (`policy.py`): an AST check with a module allowlist, no\n   `eval`/`exec`/`open`/`getattr`/…, no dunder names or attributes, no frame\n   introspection, no `global`/`nonlocal`.\n2. **Runner** (`runner.py`, stdlib only, runs inside the sandbox): restricted builtins,\n   an import hook with the same allowlist, an empty environment, rlimits (memory, CPU,\n   no file writes, no core dumps), a time limit for each call, and a per-call result\n   nonce on a duplicated stdout.\n3. **Runtime**: a `vf.Runtime`. The default is a Docker container (`python:3.11-slim`)\n   with no network, 1 GB of memory and 1 CPU. `prime` runs the code in a Prime\n   sandbox. `subprocess` runs untrusted model code **on your machine**: use it only for\n   development. The environment logs a warning when you select it.\n\nKnown limit: builder code runs in the same interpreter as the runner. Layers 1 and 2\nmake tampering with the runner hard. They do not make it impossible. Layer 3 is the\nreal boundary.\n","encoding":"utf-8","truncated":false,"total_bytes":5746},"status":null}