{"data":{"kind":"file","path":"README.md","version_id":"scva9qpi69cc03drjcw1ug0k","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3228,"modified_at":"2026-01-13T23:27:34","content_hash":"c822200c5841b030fd6a820df4c7386cf9f00ffa18b5b584cca7bf737df6ed8b"},"entries":[],"content":"# retry-toolenv\n\n### Overview\n- **Environment ID**: `retry-toolenv`\n- **Short description**: A retry wrapper for **ToolEnv** environments - wraps tool-calling environments and allows multiple retry attempts\n- **Tags**: wrapper, retry, multi-turn, tool-use, eval, train\n\n### Credits\n\nBased on `kalomaze/retry-updated`, extended to support ToolEnv.\n\n### Key Features\n- **Tool-use support**: Passes through `oai_tools` from base environment\n- **Multi-turn per attempt**: Handles multiple tool calls within each retry attempt\n- **Attempt detection**: Automatically detects when a tool trajectory completes (no more tool_calls)\n- **Async scoring**: Uses base environment rubric for scoring after each attempt\n\n### Datasets\n- **Primary dataset(s)**: Inherits from the base environment being wrapped\n- **Source links**: N/A - uses base environment dataset\n- **Split sizes**: Inherits from base environment\n\n### Task\n- **Type**: multi-turn (wrapper around ToolEnv environments)\n- **Parser**: Inherits from base environment\n- **Rubric overview**: Uses base environment rubric, applies retry penalty to final score\n\n### Quickstart\nRun an evaluation with retry wrapper on a tool-use environment:\n\n```bash\nvf-eval kalomaze/retry-toolenv -a '{\"base_env\": \"your-tool-env\", \"max_attempts\": 2}'\n```\n\nEnsure that the base environment is installed e.g. via `prime env install <env-name>`.\nYou can pass in environment arguments to the base env via `base_kwargs`.\n\n**Important**: Use with vLLM servers that have tool support enabled:\n```bash\nvllm serve MODEL --enable-auto-tool-choice --tool-call-parser hermes\n```\n\nNotes:\n- Wraps any ToolEnv\n- After each incorrect attempt, sends \"Incorrect. Try again.\"\n- Tracks inner turn count per attempt for proper max_turns handling\n\n### Environment Arguments\n\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `base_env` | str | (required) | Name of the base tool-use environment to wrap |\n| `max_attempts` | int | `3` | Maximum number of retry attempts |\n| `accept_threshold` | float | `1.0` | Score threshold to accept answer and stop retrying |\n| `retry_penalty` | float | `1.0` | Multiplicative penalty per retry (e.g., 0.5 = half points for 2nd attempt) |\n| `base_kwargs` | dict | `{}` | Additional arguments to pass to the base environment |\n\n### Metrics\n\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` | Final penalized score (raw_score * penalty^(attempts-1)) |\n| `raw_final_score` | Raw score from the final attempt before penalty |\n| `attempts_used` | Number of attempts taken (1 to max_attempts) |\n| `final_*` | All metrics from base environment prefixed with `final_` |\n\n### rl.toml Example\n\n```toml\n[[orchestrator.env]]\nid = \"kalomaze/retry-toolenv\"\nname = \"my-tool-env-retry\"\nargs = { base_env = \"your-org/tool-env\", max_attempts = 2, retry_penalty = 0.5, base_kwargs = { some_arg = \"value\" } }\n```\n\n### Difference from retry-updated\n\n| Feature | retry-updated | retry-toolenv |\n| ------- | ------------- | ------------- |\n| Tool support | No | Yes (passes oai_tools) |\n| Turns per attempt | 1 | Multiple (inner_max + 1) |\n| Env response | Always retry msg | Delegates to base env for tool responses |\n| Scoring | Sync, direct | Async, uses score_rollout |\n","encoding":"utf-8","truncated":false,"total_bytes":3228},"status":null}