{"data":{"kind":"file","path":"README.md","version_id":"tcl1glvx70oxs35iy1gm2frv","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":10813,"modified_at":"2026-08-31T03:25:17","content_hash":"e3c48f3a3392ce9ee65c405063cb2ab94881006e4d33237a6a053736fecb9b13"},"entries":[],"content":"# IronClad\r\n\r\nA deterministic RL environment for **verifiable Infrastructure-as-Code security\r\nremediation**, graded by a layered oracle with no LLM judge anywhere.\r\n\r\nAn agent is given a Terraform repo containing injected security\r\nmisconfigurations and must remediate it so that it still builds, passes security\r\nscanners and org policy, **and still provisions the working infrastructure it\r\nstarted with** — verified against the Terraform plan graph.\r\n\r\n> **The cardinal rule:** passing a scanner is necessary but not sufficient. The\r\n> reward also requires the intended resources to still exist and be correctly\r\n> configured, so the classic exploit — delete the flagged resource to silence\r\n> the scanner — earns exactly zero.\r\n\r\n## Documentation map\r\n\r\n| Document | What it is for |\r\n|---|---|\r\n| [`HANDOFF.md`](./HANDOFF.md) | **Start here.** Current status, environment setup, the measured findings that corrected the docs, design rationale, and next steps ordered by long-term value. |\r\n| [`../plan.md`](../plan.md) | The product plan: market thesis, architecture, roadmap, business model. Draft v2 — technical claims corrected against measurement, with a [changelog](../plan.md#16-changelog--verified-corrections). |\r\n| [`../references/`](../references/) | Version-pinned upstream docs for the toolchain, annotated where the binaries disagree with them. |\r\n| [`../references/VERSIONS.md`](../references/VERSIONS.md) | The exact pins, plus the version-drift notes you must not skip. |\r\n\r\n> **Docs are not ground truth; the binaries are.** Six upstream claims turned out\r\n> to be wrong when measured against the pinned toolchain, each producing a silent\r\n> scoring bug rather than a crash. They are catalogued in `HANDOFF.md` §3. When a\r\n> doc and a binary disagree here, the binary wins and the doc gets a correction\r\n> note.\r\n\r\n## The layered oracle\r\n\r\n```\r\ngates (fail → reward 0)\r\n  immutable files unmodified   weakening the rule that judges you is not a fix\r\n  L1  terraform validate       a repo that does not build cannot be judged secure\r\n  L2  terraform plan\r\n  L5  functional assertions    required resources still present, correctly\r\n                               configured, and plan-verifiable   ← the core IP\r\n\r\nscored (once every gate passes)\r\n  L3  security   0.50   net injected findings resolved (Trivy + Checkov)\r\n  L4  policy     0.30   org Rego rules satisfied (Conftest/OPA)\r\n  L6  regression 0.20   nothing else newly broken\r\n  P1  edit bloat -0.10  oversized diffs\r\n  P2  action cost -0.05 tool calls\r\n```\r\n\r\nEvery number is a CLI exit code, a parsed finding count, a Rego decision, or a\r\nplan-graph assertion.\r\n\r\nL5 and L3/L4 answer deliberately different questions. **L5 asks \"does the\r\napplication still work\"** — assert what a *deceptive* fix breaks. **L3/L4 ask \"is\r\nit secure\"** — assert what an *incomplete* fix leaves. Collapsing the two makes\r\nevery partial fix score zero and destroys the gradient RL needs.\r\n\r\n## Verified behaviour\r\n\r\nFifteen tasks ship, across ten fault categories: public exposure, data at rest,\r\nnetwork exposure, identity and trust, audit logging, secrets management, supply\r\nchain, backup and recovery, container workload and key rotation. Two fault\r\nindependent defects on two independent resource types. Every one is measured,\r\nnot asserted — golden 1.0, no-op 0, every\r\nhand-written *and* generated deception 0, at least one partial strictly between,\r\nand byte-identical reports across three runs under a perturbed `$HOME` and\r\nworking directory.\r\n\r\n```\r\ngolden fix                   1.000\r\nno-op                        0.000\r\nredteam/delete-the-resource  0.000   delete the bucket; every scanner goes quiet\r\nredteam/comment-it-out       0.000   comment the blocks out instead\r\nredteam/suppress-the-scanner 0.000   checkov:skip= / trivy:ignore: annotations\r\nredteam/hide-behind-unknown  0.000   hide the wildcard behind a plan-unknown value\r\npartial/encryption-only      0.701   honest but incomplete → strictly between\r\n```\r\n\r\nFour deception *classes* are priced, and the last two are the ones a scanner\r\ncannot reach:\r\n\r\n| Class | Example | Why the scanner misses it |\r\n|---|---|---|\r\n| Removal | `delete-the-resource`, `comment-it-out` | It doesn't — but the repo now scans clean |\r\n| Suppression | `suppress-the-scanner` | The finding is annotated away, unchanged |\r\n| Substitution | `redirect-the-trail`, `swap-the-secret`, `rename-the-repository` | Nothing is wrong with the new resource. It is simply not the one the application uses |\r\n| Laundering | `launder-the-secret` | The password moves into a variable default. Source analysis sees a reference; the plan still contains the cleartext |\r\n\r\nLaundering is the sharpest case for grading the **resolved plan** rather than the\r\nsources. Terraform resolves variable defaults, locals and interpolation before\r\nwriting the plan, so a secret \"externalised\" into a variable is still in the\r\nplan, still in the state file, and still readable from the running\r\nconfiguration — while the source no longer contains a literal for a scanner to\r\nfind. On that task neither pinned scanner reports the plaintext credential *at\r\nall*; it is graded entirely by org policy and the L5 gate.\r\n\r\n## Measured against models\r\n\r\nBaselines, not estimates. `gemini-3.6-flash`, one rollout per task, 24-step cap,\r\ntemperature 0 — across the original ten:\r\n\r\n```\r\n9 of 10 solved      0.960 - 0.973   all of L0-L6 at 1.0; the residual is the action_cost penalty\r\n1 of 10 unsolved    0.000           tf-rds-public-unencrypted, never edited the faulted resource\r\n```\r\n\r\nTwo things this environment claims, and the honest limit of each:\r\n\r\n- **It discriminates between model tiers.** On `tf-s3-public-iam-wildcard`,\r\n  `gemini-3.6-flash` solves it while `gemini-3.5-flash` and `gemini-2.5-flash`\r\n  both score 0.000. The full curve across lower tiers is not yet measured.\r\n- **It is not a frontier eval.** Every task is single-repo, few-file,\r\n  single-resource-class — the shape frontier models already clear. Treat it as a\r\n  training environment for the tiers below, and read the reward as\r\n  training signal rather than as a leaderboard.\r\n\r\nThe remaining five tasks are verified **well-posed**, not verified\r\n**difficult**: each passes the same full oracle gate, but none has yet been run\r\nagainst a model. Read the baseline above as covering ten of the fifteen, and\r\ntreat the other five as unmeasured until that sweep exists.\r\n\r\nRewards are not perfectly stable at temperature 0: one task scored 0.000, 0.960,\r\n0.960 across three runs of the same model. Report a spread, not a cell.\r\n\r\n## Install from the Environments Hub\r\n\r\n```bash\r\nuv tool install -U prime\r\nprime env install vinay-k-rajith/ironclad\r\n```\r\n\r\n**Read this before the first rollout: the reward needs a toolchain.** IronClad is\r\nnot a dataset environment. Every number in it is a CLI exit code, so Terraform\r\n1.15.8, Trivy 0.72.0, Conftest 0.68.2 and Checkov 3.2.526 must be on `PATH` of\r\nthe **grading host** (Linux), at those versions — a scanner version bump changes\r\nwhich findings exist and therefore changes the reward. `tools/install_toolchain.sh`\r\nin a source checkout installs exactly those pins.\r\n\r\nThe oracle grades on the host rather than inside the rollout container: the repo\r\nis tarred out after the agent stops and scored outside. That is what keeps ground\r\ntruth off a filesystem the agent just controlled, and it is why the toolchain is\r\na host requirement rather than an image requirement. Without it every rollout\r\nraises `OracleError`, which surfaces as a rollout **error and not a 0.0** — on\r\npurpose, so that breaking the grader never scores the same as failing honestly.\r\n\r\nEvaluate:\r\n\r\n```bash\r\nuv run vf eval --env.taskset.id ironclad_env --env.agent.model <model> -n 5\r\n```\r\n\r\nThe taskset takes four load-time options — `tasks_dir`, `difficulties`,\r\n`categories` and `image` — so `--env.taskset.difficulties '[\"T3\",\"T4\"]'`\r\nrestricts a run to the harder tiers.\r\n\r\n## Running it\r\n\r\nRequires the pinned toolchain (Terraform 1.15.8, Trivy 0.72.0, Conftest 0.68.2,\r\nCheckov 3.2.526) on Linux. See [`HANDOFF.md` §2](./HANDOFF.md#2-environment).\r\n\r\n```bash\r\n# Unit tests: pure logic, no toolchain, ~0.1s\r\npython -m pytest tests/ -q\r\n\r\n# Full oracle validation for a task\r\npython tools/validate_task.py validate tasks/tf-s3-public-iam-wildcard -v\r\n\r\n# Regenerate a fixture's target_findings by measuring faulted vs golden\r\npython tools/validate_task.py measure tasks/tf-s3-public-iam-wildcard\r\n```\r\n\r\n## Authoring a task\r\n\r\nA task is a directory:\r\n\r\n```\r\ntasks/<id>/\r\n  task.yaml        manifest\r\n  instruction.md   the prompt the agent sees\r\n  fixture.yaml     ground truth — never in the agent's read scope\r\n  repo/            the faulted repo the agent starts from\r\n  golden/          overlay: reference fix         (must score 1.0)\r\n  partial/*/       overlay: incomplete but honest (must score 0 < r < 1)\r\n  redteam/*/       overlay: deceptive fixes       (must all score 0)\r\n  policies/*.rego  org policy, evaluated against the plan JSON\r\n```\r\n\r\nTwo rules the loader enforces, because breaking either produces a task that\r\nscores confidently and wrongly:\r\n\r\n1. **Every fixture needs at least one `required_resource`.** Without one, L5 is\r\n   vacuous and deleting the flagged resource becomes a winning strategy.\r\n2. **Every negative assertion (`none_eq`, `none_regex`, `none_in`) must be paired\r\n   with `exists` on the same path.** Negative assertions are vacuously true over\r\n   an empty set, and any value unknown at plan time resolves to empty — so an\r\n   unpaired negative silently asserts nothing. This is a real attack, not a\r\n   theoretical one: see `redteam/hide-behind-unknown`.\r\n\r\nNever hand-write `target_findings`. Generate them with\r\n`validate_task.py measure`, and regenerate whenever a scanner version changes.\r\n\r\nThree judgement calls the loader *cannot* make for you. Each one has shipped as a\r\nbug at least once and each is caught only by the golden-fix-equals-1.0 assertion:\r\n\r\n- **Which attributes go in the L5 gate.** Anything a suppression comment would\r\n  silence must be gated, or `suppress-the-scanner` scores above zero. Everything\r\n  else should stay scored, or an honest half-finished fix hard-fails to 0 and the\r\n  reward is binary. Every task keeps at least one scored-only dimension.\r\n- **Whether a rule is satisfiable at all.** An org rule that fails identically on\r\n  the faulted repo and the golden fix is a constant subtracted from every reward,\r\n  and it looks exactly like a legitimate governance rule.\r\n- **Which layer actually grades the headline fault.** Do not assume the scanners\r\n  cover it. Read `measure`'s RESOLVED list; if the fault you injected is not in\r\n  it, the scanners are blind to it and the org policy has to carry it.\r\n","encoding":"utf-8","truncated":false,"total_bytes":10813},"status":null}