{"data":{"kind":"file","path":"README.md","version_id":"cqntjh4d2zsst1t74ru08h5y","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":8572,"modified_at":"2026-08-25T13:09:40.143000","content_hash":"acd97ed9be907ade6dda5d112cedff65557c12961c1963c577a532b6c39d4f3f"},"entries":[],"content":"# causal-gym\n\ncausal-gym is a native Verifiers v1 taskset for causal inference from\nobservational tabular data. Each task generates a hidden structural causal\nmodel, stages only data.csv and a data dictionary, and accepts one typed JSON\nsubmission through submit_answer.\n\n## Design\n\nThe shipped families are ate, adjustment, and null_detect. The generator rejects\ntasks that fail the SCM identification certificate or the solvability check on\nthe exact staged frame.\n\nGeneration starts from a stable public task_id. A keyed SHA-256 derivation\ncreates independent scm, data, messiness, naming, and prompt random streams.\nTrain IDs use the train namespace; the frozen split uses the eval-v1 family\nnamespace, making the\nnamespaces disjoint by construction. The default key is the public\nreproducibility key in rng_keys/derive.py. CAUSAL_GYM_KEY (hex) overrides\n--env.taskset.task.env-key, which overrides that default.\nTraining difficulty-tier sample sizes are deterministically jittered by up to\n50% on a task-keyed stream. Adjacent tier ranges overlap, so row count does not\nuniquely identify the tier, but the distributions remain distinct and row count\nis still informative. Eval-v1 instead uses the exact materialized row counts\nbaked into its frozen manifest; train-stream obfuscation does not apply to that\npublic split.\n\nTasks require a container, use /workspace, and request framework-only network\naccess. They use the stock `python:3.11-slim` image. Before network isolation,\nsetup installs pinned NumPy, pandas, SciPy, statsmodels, and scikit-learn wheels,\nthen writes only data.csv and README.md. No custom registry image is required.\n\n## Identification assumptions\n\nThe requested estimand is the population average treatment effect. Every task\nis certified as backdoor-identifiable from observed pre-treatment covariates.\nThe agent-visible data dictionary labels every column as pre- or post-treatment\nand states conditional ignorability given the observed pre-treatment\ncovariates, consistency, and positivity. Adjustment answers may contain only\nknown pre-treatment columns and must d-separate treatment and outcome in the\nbackdoor graph.\n\n## Reward\n\nAnswers report a 90% interval. Let W be the calibrated 88% Winkler loss and let\nr be the matching reference width from the staged adjusted-OLS standard error.\nThe estimand loss is L = W + |estimate - truth| + 2 × max(0, r -\ninterval_width). For a correct, centered interval this is uniquely minimized at\nwidth r, so narrower intervals lose reward. The observable scale s is the\nnan-ignoring population standard deviation of the staged outcome column,\nfloored at 1e-6. Estimand reward is 1 - L / s. It is intentionally unbounded\nbelow; RL training should clip or normalize rewards.\nThe display-only `clipped_reward` metric reports max(-1, reward) without changing\nthe scored reward.\n\n- ate: estimand reward.\n- adjustment: 0.7 × estimand + 0.3 × adjustment_validity.\n- null_detect: 0.9 × estimand + 0.1 × prior-normalized Brier skill for the\n  submitted effect probability. The disclosed effect prior is 13/20 (7/20\n  structural null, 7/20 hard non-null, 6/20 wider non-null), so a prior-only\n  probability forecast has expected classification skill zero.\n\nThe null-detection generator uses keyed 20-task blocks with 7 structural nulls,\n7 hard non-nulls (0.05 <= |ATE| <= 0.25), and 6 wider non-nulls. Invalid\nsubmissions return only schema validation errors. Retries cost turns; the first\nvalid submission is final.\n\n### Anti-hack audit\n\nThe assembled-scorer audit uses 300 successfully materialized, staged tasks per\nfamily. The honest reference deduplicates rows, removes robust outliers, uses\nREADME-declared pre-treatment covariates, and fits OLS. Full results are in\n[`causal_gym/audit_report_v6.json`](causal_gym/audit_report_v6.json).\n\n| Family | Honest reward | Interval coverage |\n| --- | ---: | ---: |\n| ATE | 0.804614 | 0.906667 |\n| Adjustment | 0.859952 | 0.900000 |\n| Null detection | 0.775130 | 0.900000 |\n\nThe honest null-detection reference reports probability 0.25 when its interval\ncontains zero and 0.95 otherwise. Across all three families, its mean reward was\n0.813232 and interval coverage was 0.902222. The ATE width sweep peaked at 0.90\nnominal coverage (0.906667 empirical). On structural nulls, the 1%-scale-width\ninterval scored 0.813290, strictly below the honest reference-width interval at\n0.902268. All\n900 audit tasks materialized with no skipped IDs.\n\n## Threat model and anti-hack measures\n\nderive_streams prevents sequential public seeds from serving as a direct oracle\nlookup and separates SCM, data, messiness, naming, and prompt randomness. It\ndoes not make a published key secret: DEFAULT_EVAL_KEY is intentionally public\nso frozen published splits can be reproduced exactly. A known key plus a small\nenumerable task-ID space permits exact dataset fingerprinting and truth recovery.\n\nSecret-key evaluation uses a private CAUSAL_GYM_KEY and task IDs drawn from the\nfull derivation domain. The key, generator package, and task manifest stay\nhost-side. Runtime hacking is prevented by sandbox isolation: the agent image\nhas no generator package and no network. Training should use fresh-key infinite\nstreams, keeping train IDs in the train namespace and evaluation IDs in the\neval-v1 namespace; fresh keys mitigate memorization and contamination.\n\nPublished-split contamination is inherent to open synthetic benchmarks and\ncannot be cryptographically prevented once generator inputs are disclosed.\nExact PRNG/source replay is therefore prohibited by policy and detectable:\nreplayed answers correlate with hidden truth beyond the information limit of\nthe agent-visible data. Keying supplements sandboxing, audits, an observable\nreward scale, strict schemas, validity-based adjustment scoring, structural\nnulls, and fresh held-out streams; it is not a substitute for them.\n\nDo not publish normal trace/config artifacts from a secret-key evaluation:\ntask IDs are trace data and a configured key may be present in resolved config.\nPrefer the CAUSAL_GYM_KEY environment variable and publish only redacted\naggregate results.\n\n## Limitations\n\n- The open eval-v1 key and manifest provide reproducibility, not secrecy.\n- Solvability is certified against adjusted OLS on staged generated data; real\n  model difficulty still needs separate calibration.\n- Each fresh sandbox installs the pinned analyst wheels during trusted setup.\n- Confidence intervals are scored per task; empirical calibration still\n  requires evaluation over many tasks.\n\n## Baseline results\n\nReal Mac-side Docker rollouts on commit `de631d50`:\n\n| Model | Provider | Sample | n | Mean reward | ATE | Adjustment | Null detection | Coverage |\n| --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |\n| `x-preview-f-free` | keyless 0xalpha Zen | shuffled `eval-v1` | 25 | 0.765977 | 0.659610 | 0.890375 | 0.761242 | 0.840000 |\n| `openai/gpt-4.1-mini` | OpenRouter | train prefix, partial | 5 | -0.611686 | -2.432305 | 0.798327 | 0.209524 | 0.400000 |\n\n0xalpha completed 25/25 with no final errors; four episodes exceeded the\n600-second agent budget and recovered on their first retry (16% episode retry\nincidence), while the Zen proxy needed 0 retries across 241 successful calls.\nThe GPT run retained 5/20 planned episodes before funding stopped it; its logs\ncontain two HTTP 402 failures among nine rollout starts. The samples are not\nmatched, so the comparison is a directional difficulty check rather than a\ncontrolled model ranking.\n\n## Roadmap\n\nCATE, mediation, and observational-versus-RCT tasks remain disabled until each\nhas its own typed estimand, staged evidence, identification contract, oracle,\nschema, reward, validator, and baseline audit.\n\n## Evaluate\n\nValidate the package TOML and run:\n\n    uv run python -c 'import tomllib; tomllib.load(open(\"environments/causal_gym/pyproject.toml\", \"rb\"))'\n    uv run eval causal_gym -n 1 --env.agent.runtime.type docker\n    uv run eval causal_gym -n 1 --env.taskset.split eval-v1 --env.agent.runtime.type docker\n\nUse -n 500 to run the complete frozen manifest.\n\nUse a fresh private key for training:\n\n    CAUSAL_GYM_KEY=\"$(openssl rand -hex 32)\" uv run eval causal_gym -n 1 --env.agent.runtime.type docker\n\nRun package checks from the repository root:\n\n    uv run pytest environments/causal_gym/tests/ -q\n    uv run ruff check environments/causal_gym\n\n## Availability\n\nPublished on the Prime Intellect Environments Hub:\n\n    prime env install ulcompute/causal-gym\n\nHub page: <https://app.primeintellect.ai/dashboard/environments/ulcompute/causal-gym>\n","encoding":"utf-8","truncated":false,"total_bytes":8572},"status":null}