{"data":{"kind":"file","path":"README.md","version_id":"p01sryendvjmjfhq6rwjkpzd","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":7052,"modified_at":"2026-11-10T14:10:52.626000","content_hash":"e148b6f06e91365fd12e6841f599d21efaa45bda3699ce2b5ef7966d1814b392"},"entries":[],"content":"# Long-Horizon Filesystem + Memory Environment\n\n**Prime Intellect / Verifiers v1 — taskset package, v0.1.0**\n\nA coding-agent RL/evaluation taskset with realistic file-based incidents, durable handoff memory, a destructive (document-removing) checkpoint, and functional tests staged **only in an isolated verifier**. Designed for Prime Sandboxes or Docker, with the general-purpose **bash** coding harness. Does not introduce environment-specific MCP tools.\n\n> **Validation:** 20 local tests passed with Verifiers 0.4.1; both scenario gold demonstrations scored 1.0. Taskset discovery, train/eval construction, evaluation configurations, and installed-wheel imports were checked. These are deterministic checks, not a real-model evaluation. No real-model sandbox rollout has been performed.\n\n## Scenario families\n\n| Family | Work to complete | Hidden functional checks |\n|---|---|---|\n| `replay` | Diagnose JSONL event ledger; fix validation, deduplication and chronological balance replay | Last-valid duplicate, `set` order, timezone offsets, malformed records, ties, negative balances, randomized logs |\n| `config` | Fix three-layer deployment settings, precedence and type coercion | Invalid keys, canonical integer strings, booleans, immutability, boundaries and randomized layer combinations |\n\nEach seed produces different incident files and an on-call handoff token. Train and eval use disjoint seed ranges, with **8 seeds × 2 families = 16 tasks per split** by default. These are *procedural variations*, not 16 independent human-authored problems.\n\n## Agent workflow\n\n1. Navigate to `/workspace/incident_lab`, inspect sources, public tests, input data and `data/ops_notes.md`.\n2. Persist critical incident details and the unique handoff token in `.agent/memory.md`.\n3. Run `python3 tools/checkpoint.py` from the workspace root, which deletes the original operations note and adds `data/stage2.md`.\n4. Continue solving from the remaining files and durable memory. Repair code, write regression tests, and update README.\n5. Scoring happens in a *new* sandbox, after the source directory is restored and the private grader is written only into the verifier sandbox.\n\nThis evaluates file-based memory persistence and checkpoint handoff **within one multi-turn rollout**. It does **not** yet force a model context-window reset or prove that the agent actually read its memory; the checkpoint and memory contents are evidence, not cryptographic attestation.\n\n## Reward (0.0 to 1.0)\n\n| Component | Weight | Evaluation |\n|---|---:|---|\n| Functional correctness | 72% | 24+ private cases per scenario (incl. randomized property-like cases) |\n| Durable memory | 10% | Token + scenario-specific engineering facts in `.agent/memory.md` |\n| Checkpoint | 6% | Checkpoint marker + removed policy file + stage-2 note |\n| Regression tests | 8% | Substantive regression test file; unittest discovery succeeds |\n| README | 4% | API, critical rules, and test command documented |\n\nAll rewards are deterministic; a free-form response cannot earn functional credit. Private grading requires isolated scoring (enabled by default) and a container runtime (`prime` or `docker`). Use an image with `python3` available. No network access is needed inside the task. Private grading code is included in the taskset distribution but must not be copied into the solver workspace; it is not a secret to someone who downloads the distribution. Seed variation therefore measures robustness rather than adversarial benchmark secrecy.\n\n## Installation\n\nFrom this directory:\n\n```bash\nuv sync --extra dev\nuv run pytest -q\n```\n\nFor a Prime Lab / prime-rl workspace, install this package into that environment:\n\n```bash\nuv pip install -e /path/to/longhorizon-fs-memory-env\n```\n\nNewest Verifiers release (as checked November 10, 2026) uses `vf-run` for quick in-process tests; full eval/training is supported by **prime-rl** `eval` runner. Use the evaluated CLI's `--help` before launching, as CLI flags have recently changed. Example prime-rl run (after copying `configs/eval-prime.toml` into your prime-rl workspace and setting credentials):\n\n```bash\nuv run eval @ configs/eval-prime.toml\n```\n\nThe included `configs/eval-docker.toml` switches agent/verifier to Docker, while `configs/eval-prime.toml` uses Prime Sandboxes. A model identifier in those configs is an example; replace it with an accessible model in your account.\n\nEach task declares `scoring.mode = \"isolated\"`, using the Verifiers 0.4.1 scoring lifecycle to call `stage_verifier` in a fresh container after artifacts are restored. The standard single-agent environment is used; no `isolated-verifier` environment plugin is required.\n\n## Offline smoke tests and manual demonstrations\n\n```bash\nuv run pytest tests -q\nuv run python scripts/smoke_local.py --family replay --seed 3\nuv run python scripts/smoke_local.py --family config --seed 3\n```\n\n`smoke_local.py` populates a **temporary local workspace**, runs the grader on the unsolved baseline, and prints the score. This is not a model rollout. Use the unit tests to verify the oracle and the no-op lower bound.\n\n## Recommended release gates\n\n- Validate at least 50 train and 50 eval rollouts with a strong harness and report success rates, reward distribution, average turns and wall-clock cost.\n- Test with no-op and trivial boilerplate agents to discover reward hacking; compare with a gold solution and require the no-op reward to remain low.\n- Run prompt-injection, symlink, memory-spoofing, checkpoint-bypass, and grader-file tampering audits.\n- Extend dataset families to 5–10 independent tasks, each with parametrized variants; a two-family v0.1 is a credible **prototype**, not evidence of production benchmark breadth.\n- Run a real Prime sandbox model evaluation before making benchmark performance claims. Publication alone does not establish model performance or hosted CI success.\n\n## Structure\n\n```text\nlonghorizon_fs_memory/\n  __init__.py       # exports LongHorizonTaskset\n  taskset.py        # Verifiers TaskData/Task/Taskset, setup, score\n  fixtures.py       # deterministic workspaces and checkpoint script\n  grading.py        # withheld-from-agent private deterministic grader\nconfigs/\n  eval-prime.toml\n  eval-docker.toml\nscripts/\n  smoke_local.py\n  gold_local.py\n tests/\n  test_fixtures.py\n  test_grading.py\n```\n\n## Source links\n\n- [Prime Verifiers v1 taskset guide](https://github.com/PrimeIntellect-ai/verifiers/blob/main/docs/v1/tasksets.md)\n- [Isolated verifier contract](https://github.com/PrimeIntellect-ai/verifiers/blob/main/docs/v1/env.md)\n- [Latest Verifiers release](https://github.com/PrimeIntellect-ai/verifiers/releases)\n- [Prime Envs authoring guidance](https://github.com/PrimeIntellect-ai/prime-envs/blob/main/AGENTS.md)\n\n## Fast publish path (skip testing)\n\nWhen deliberately skipping all local tests and agent rollouts, follow\n[DEPLOY_NO_TESTS.md](DEPLOY_NO_TESTS.md). From the package root:\n\n```bash\nprime env push --path .\n```\n\nThis *attempts publication*; it does not validate correctness or guarantee Hub approval.\n","encoding":"utf-8","truncated":false,"total_bytes":7052},"status":null}