{"data":{"kind":"file","path":"README.md","version_id":"fubi5nqu69t57o0mc9egrfyq","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3550,"modified_at":"2026-10-14T12:24:54.249000","content_hash":"40cc7fb6c3d71d6263738558b10acdda36875e250148794e6b35f889f0b291a3"},"entries":[],"content":"# API Contract Repair\n\nThis Verifiers v1 package contains one coding task, `pagination-001`. The agent repairs `list_items` in `/workspace/catalog.py`: it must filter items by category before taking a one-based page, preserve order, return the total matching count, and reject invalid page or page size values.\n\nThe agent sees a deliberately buggy `catalog.py`, a public `unittest` file, and a prompt describing the contract. The `bash` harness runs in Docker. After the agent finishes, `isolated-verifier` copies only the edited `catalog.py` into a separate verifier container. The binary `held_out_tests` reward is `1.0` when every scoring test passes and `0.0` otherwise. `vf-validate` also checks that the known fix passes and the original bug fails.\n\n## Run locally\n\nFrom the RL-Agents repository root, configure the ignored `.env` with a reachable OpenAI-compatible model endpoint and exact model ID, then run:\n\n```bash\n./scripts/local.sh validate api-contract-repair\n./scripts/local.sh models\n./scripts/local.sh dry-run api-contract-repair\n./scripts/local.sh eval api-contract-repair\n```\n\nValidation requires Docker but no model. The model check and evaluation require `.env`. Local evaluations use `push = false` and write private traces under ignored `outputs/`; they do not upload to Prime or train the model.\n\n## Publish the environment and one evaluation\n\nFrom the repository root, after reviewing the package source and a completed local run:\n\n```bash\nprime whoami\nprime env push --path environments/api_contract_repair \\\n  --runtime v1 --visibility PUBLIC\nprime env status akashdeepjassal/api-contract-repair --output json --plain\n```\n\nFor a new environment in your own account, leave out `--owner`; the CLI uses it for collaborator pushes to an existing owner. A message about the earlier range-repair upstream comes from the repository's root `.prime` metadata and does not change this package's name. Publishing the environment uploads this package, including the scoring tests. To submit a **separate** public evaluation, set `RUN` to the exact completed local output directory and `MODEL_ID` to the model used for that run:\n\n```bash\nRUN='outputs/replace-with-your-completed-run'\nMODEL_ID='replace-with-your-model-id'\npython3 scripts/prepare_prime_upload.py \\\n  \"$RUN/traces.jsonl\" \\\n  outputs/api-contract-repair-upload \\\n  --model \"$MODEL_ID\" \\\n  --env akashdeepjassal/api-contract-repair\ncat outputs/api-contract-repair-upload/metadata.json\ncat outputs/api-contract-repair-upload/results.jsonl\nprime eval push outputs/api-contract-repair-upload \\\n  --env akashdeepjassal/api-contract-repair --public --output json --plain\nprime eval get '<evaluation-id-from-push>' --output json --plain\nprime eval samples '<evaluation-id-from-push>' --output json --plain\n```\n\nInspect both generated files before `prime eval push`: the score-only payload should contain no prompts, model messages, endpoint URLs, or keys. The upload directory must be new and under ignored `outputs/`. In the final `prime eval get` response, check the score, `is_public`, and `show_on_leaderboard` separately. A public evaluation is not guaranteed a leaderboard entry. Neither publishing step trains the model.\n\n## Public-source limitation\n\nScoring tests are kept out of the agent's workspace during a local attempt, but their definitions are included in this package. Anyone who inspects a publicly published package can read them. This one-task environment is a learning example and local smoke test, not a cheat-resistant benchmark or a reliability estimate.\n","encoding":"utf-8","truncated":false,"total_bytes":3550},"status":null}