{"data":{"kind":"file","path":"README.md","version_id":"vpdupo8wqvv8639y0f1355zl","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2060,"modified_at":"2026-10-31T17:49:27.493000","content_hash":"b3d3969dfd3dc8db706efd606a50ffd822fbc9b185a7b4b80cfeeeee8d95a540"},"entries":[],"content":"# faultline-vf\n\n[faultline](https://github.com/namanxdev/faultline) as a verifiers v1 taskset. It is an\nRL training and evaluation environment for tool-using agents whose MCP tools **drift**\nwhile they work: a parameter is renamed, a description changes, and the agent must\nnotice and adapt.\n\n- **Task.** One faultline task under one drift. Each rollout gets a fresh SQLite\n  database and its own MCP server process, which serves the drifted tools.\n- **Reward.** `solved` is the grade of the final database state against the policy\n  text. The grader never reads the transcript. Optional bounded shaping is available\n  with `--env.taskset.task.shaped true`.\n- **Gold check.** The `validate` hook runs the oracle agent, which must solve every\n  task under every drift.\n- **Splits.** `train` (default), `dev`, and `test`, set with `--taskset.split`.\n\n## Usage\n\nverifiers imports `fcntl`, so this runs on Linux and macOS, not Windows.\n\n```sh\nprime env install naman411/faultline-vf\nuv run vf-validate faultline-vf --taskset.split dev --select.limit 26 --runtime.type subprocess\n```\n\nThe package depends on `faultline`, which is not on a package index. To run from\nsource, clone the repository and work in `environments/faultline_vf`:\n\n```sh\ngit clone https://github.com/namanxdev/faultline\ncd faultline/environments/faultline_vf\nuv sync\nuv run vf-validate faultline-vf --taskset.split dev --select.limit 26 --runtime.type subprocess\nuv run vf-eval @ configs/smoke.toml -n 4 -r 1 --no-push   # needs OPENROUTER_API_KEY\n```\n\n## Recorded results\n\n- `vf-validate` on `dev`, 26 tasks: gold and noop were both valid 26/26.\n- 4 rollouts of Haiku 5.5 with the `null` harness (task tools only, no shell) all\n  scored reward 1.00, for $0.0065. This is a smoke test, not a benchmark.\n\n## Notes\n\nThe package overrides `mcp` to 2.0.0, which verifiers 0.4.0 pins. faultline itself pins\n2.3.0. The taskset uses only faultline's tools, drifts, tasks, and grader, which pass\nfaultline's suite on mcp 2.0.0 except for faultline's own stdio runner, which this\npackage does not use.\n","encoding":"utf-8","truncated":false,"total_bytes":2060},"status":null}