{"data":{"kind":"file","path":"README.md","version_id":"hkdm2m8hbxcfb1k8uxen0vkk","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":9254,"modified_at":"2026-10-28T20:27:46.859000","content_hash":"043939083d3d5a8a4cfdc237de2d8168736f34c552f9ef57c3ccf1936d6d6b5f"},"entries":[],"content":"# travel-agent\n\nA multi-turn RL environment for an AI travel agent, packaged for Prime Intellect's\n[`verifiers`](https://github.com/PrimeIntellect-ai/verifiers) framework and the\nEnvironments Hub.\n\nThe model plays a travel agent operating a booking console for a simulated client:\nit searches and books flights, hotels and activities, asks the client about hidden\npreferences, discloses the total, presents the plan and fixes the client's feedback,\nconfirms, and then (on harder curriculum levels) lives through the trip day by day,\nrebooking around cancelled flights, overbooked hotels and weather closures.\nEpisodes last up to 60 steps.\n\nThe simulator is `travel_env.TravelAgentEnv` (vendored here, source of truth is the\nrepository's top-level `travel_env/`). It exposes an *enumerated* action space: each\nstep lists every legal, fully-bound action (`obs[\"legal_actions\"]`, each with an `id`\nand a `label`), so the model never has to invent flight ids or hotel names; it picks\nan index. Incoherent actions are unrepresentable rather than merely penalised.\n\n## Chat protocol\n\n- **System prompt** explains the console, the scoring rules and the one-integer reply format.\n- **Each user turn** contains: the result of the previous action, `env.render()`\n  (phase, budget, request, revealed preferences, bookings, pending feedback, open\n  disruptions), the results of the last search (one line per flight/hotel/activity),\n  and the numbered `LEGAL ACTIONS` list.\n- **Each assistant turn** must be exactly one integer, the action id. The parser also\n  accepts `<action>N</action>`, `Action: N` / `answer: N`, and falls back to the last\n  standalone integer in the reply (`<think>...</think>` blocks are stripped first).\n  An unparseable or out-of-range reply is passed to the simulator as an out-of-range\n  index, which the simulator treats as an invalid action: a no-op that costs\n  `invalid_action_penalty` and a step.\n- The episode ends when the simulator reports `terminated`/`truncated`; the final\n  user message is an `EPISODE OVER (...)` summary.\n\nA full 60-step episode accumulates roughly 3.5K characters per turn, i.e. about\n50K tokens of context by the end. Use a model with at least a 64K context window, or\npass a smaller `max_turns` for cheap smoke evals.\n\n## Reward and metrics\n\nThe rubric's single weighted reward is the simulator's terminal `total`, clipped to\n`[0, 1]` for training (the raw range is `[-0.05, 1]`):\n\n```\ntotal = gate * (0.45*satisfaction + 0.20*budget + 0.20*coherence + 0.15*recovery) - time_penalty*steps/60\ngate  = 1.0 if every leg is flown and every night housed, else 0.3 * completeness\n```\n\nWeight-0 metric functions are attached so they show up in `vf-eval` tables:\n`travel_total_raw`, `travel_satisfaction`, `travel_budget`, `travel_coherence`,\n`travel_recovery`, `travel_completeness`, `travel_gate`, `travel_steps`,\n`travel_invalid_actions`, `travel_unparsed_replies`.\n\nIf verifiers cuts a rollout short (its own `max_turns`, default 64 > the simulator's\n60-step cap, so this should not happen) the reward function force-finishes the\nsimulator so a terminal score still exists (`terminal[\"reason\"] == \"rollout_cut_short\"`).\n\n## `load_environment`\n\n```python\nimport verifiers as vf\nenv = vf.load_environment(\"travel-agent\", level=1, num_train_examples=200, num_eval_examples=50, seed=0)\n```\n\n| arg | default | meaning |\n| --- | --- | --- |\n| `level` | `1` | curriculum level `0..3`: `L0_solo_simple`, `L1_couple_feedback`, `L2_tight_budget_disruptions`, `L3_multicity_chaos` |\n| `num_train_examples` | `200` | rows with seeds `seed .. seed+N-1` |\n| `num_eval_examples` | `50` | rows with seeds offset by `eval_seed_offset` (default `100_000`) so eval worlds never coincide with train worlds |\n| `seed` | `0` | base seed |\n| `max_turns` | `64` | verifiers turn cap; keep it above 60 so the simulator's own terminal reward is always reached |\n| `config_path` | auto | path to a `world_default.json`-style config (destinations, pricing, disruption rates, reward weights) |\n| `system_prompt` | built-in | override the system prompt |\n\nDataset rows are `{\"question\": <first observation>, \"answer\": \"\", \"info\": {\"level\": L, \"seed\": s}}`.\nOne `TravelAgentEnv` is created lazily per rollout from `info` (so rollouts are deterministic\nand independent) and kept in a module-level registry keyed by a per-rollout id stored in\n`state`, which keeps `state` JSON-serialisable.\n\n## Layout\n\n```\nenvironments/travel_agent/\n  pyproject.toml      # name \"travel-agent\", hatchling, depends on verifiers\n  travel_agent.py     # load_environment(), TravelAgentMultiTurnEnv, ActionIdParser, rubric\n  README.md\n  test_smoke.py       # deterministic, no-LLM smoke test\n  travel_env/         # VENDORED copy of ../../travel_env  (do not edit here)\n  configs/            # VENDORED copy of ../../configs     (do not edit here)\n  VENDORED.md         # written by the sync script\n```\n\n`travel_env/` and `configs/` are copies made by `scripts/sync_prime_env.sh` (run it from\nthe repository root after changing the simulator). The wheel force-includes `configs/` as\n`travel_env/configs/` so the installed package is self-contained; `travel_agent.py` looks\nfor the config in both places.\n\n## Local install and smoke test\n\n```bash\n# from the repository root\nuv pip install -e environments/travel_agent        # or: vf-install travel-agent  (looks in ./environments)\npython environments/travel_agent/test_smoke.py     # or: pytest -q environments/travel_agent/test_smoke.py\n```\n\nThe smoke test drives the `MultiTurnEnv` through `setup_state` / `is_completed` /\n`env_response` exactly as `verifiers`' rollout loop does, with scripted policies\n(\"always reply 0\", garbage replies, confirm-immediately, and the repository's scripted\n`HeuristicPolicy` used as an oracle), checks determinism, the invalid-action path, early\ntermination, the force-finish path, and that the rubric functions return the terminal\nscores. It needs no model and no network. If `verifiers` is not importable it installs a\nminimal stand-in with the same method surface and says so in its first output line.\n\n## Local eval (any OpenAI-compatible endpoint)\n\n```bash\nexport OPENAI_API_KEY=...            # or OPENAI_BASE_URL for a local vLLM server\nvf-eval travel-agent -m gpt-4.1-mini -n 20 -r 1 -a '{\"level\": 1}'\nvf-eval travel-agent -m gpt-4.1-mini -n 20 -r 1 -a '{\"level\": 2, \"num_eval_examples\": 20}' -s   # -s saves results\n```\n\n## Push to the Hub\n\nRequires the Prime CLI (`uv tool install prime` or `pip install prime`) and an API key\nfrom <https://app.primeintellect.ai>. Confirm the exact subcommands with\n`prime env --help` first; the names below are the documented ones at the time of writing.\n\n```bash\nprime login                                    # paste your API key (stored in ~/.prime)\nscripts/sync_prime_env.sh                      # from the repo root: refresh the vendored simulator\ncd environments/travel_agent\npython test_smoke.py                           # must print SMOKE TEST PASSED\nprime env --help                               # confirm `push` and its flags\nprime env push                                 # builds the wheel with uv and uploads it\n# optional: prime env push --visibility PUBLIC\n```\n\nAfter the push the environment is installable anywhere with\n`prime env install <your-handle>/travel-agent` (and importable as `travel_agent`).\n\n## Run evals on Prime\n\n```bash\nprime login\nprime env install <your-handle>/travel-agent\n# verifiers' eval CLI, pointed at a model served through Prime Inference:\nexport OPENAI_API_KEY=$PRIME_API_KEY\nexport OPENAI_BASE_URL=https://api.pinference.ai/api/v1      # confirm in the Prime dashboard\nvf-eval travel-agent -m <model> -n 20 -r 1 -a '{\"level\": 1}'\n# if the CLI offers hosted evals, use it instead of running locally:\nprime eval --help\nprime eval run travel-agent -m <model> -n 20 -r 1         # exact syntax per `prime eval --help`\n```\n\nSuggested first model: a small instruction-tuned model such as `Qwen/Qwen3-4B-Instruct`\n(or whatever small model Prime Inference lists). Expect low scores from untrained models:\nthe itinerary must be *complete* before the gate opens, so a model that confirms early or\nkeeps searching scores ~0; the scripted heuristic scores about 0.5 on level 1.\n\n## Training with prime-rl\n\n```toml\n# orchestrator section of a prime-rl config\n[[orchestrator.env]]\nid = \"travel-agent\"\nargs = { level = 1, num_train_examples = 2000, num_eval_examples = 100, seed = 0 }\n```\n\nStart on level 0/1 and raise `level` as `travel_completeness` approaches 1.0; level 2\nadds tight budgets and disruptions (recovery matters), level 3 is multi-city.\n\n## verifiers API compatibility\n\n- Requires `verifiers>=0.1.3` (async `MultiTurnEnv` with `setup_state`, `is_completed(messages, state)`\n  and `env_response(messages, state)`).\n- `env_response` returned `tuple[Messages, State]` up to 0.1.5 and returns `Messages` only\n  (state mutated in place) in newer releases. `travel_agent.py` inspects the base class's\n  return annotation at import time (falling back to the package version) and returns the\n  right shape; `_RETURNS_TUPLE` tells you which was detected.\n- Reward functions take `state` by keyword (`def travel_reward(state, **kwargs)`), which\n  verifiers supports in all of these versions.\n","encoding":"utf-8","truncated":false,"total_bytes":9254},"status":null}