{"data":{"kind":"file","path":"README.md","version_id":"vezakfhk1g1zn3o0jkxdtrkz","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":10549,"modified_at":"2026-07-14T11:46:50.464000","content_hash":"20223550ce27433eed0b7762a94715f9c095f3036cbc0344bddb40e251e08c08"},"entries":[],"content":"# uav-operator\n\n`uav-operator` is a multi-turn Prime Intellect / Verifiers environment where\nan LLM acts as the remote pilot in command for small UAS missions over the San\nFrancisco Bay Area.\n\nDrones already fly themselves. Autopilots hold trajectories, execute return to\nlaunch, and enforce simple failsafes. This environment tests the layer above\nthat: ops-center judgment under shifting wind, pop-up flight restrictions,\nbattery anomalies, lost-link events, and mission updates. The simulator owns\nthe aircraft and every reward is computed from logged simulator state, not from\nthe model's prose.\n\n![UAV ops console: Laguna handles four composed events and lands safely](assets/renders/day7/hero-treasure-island-safe-response.gif)\n\n### Overview\n- **Environment ID**: `uav-operator`\n- **Short description**: Multi-turn UAV operations environment with\n  physics-derived rewards and tool-based operator decisions.\n- **Tags**: `uav`, `drone-operations`, `multi-turn`, `tool-use`, `train`,\n  `eval`\n- **Status**: Day 7 soft-launch release candidate. The offline renderer, five\n  fixed Laguna T3 clips, hero GIF, frozen rollout artifacts, and `0.1.1`\n  package candidate are complete. Public Hub publication remains gated on the\n  cold-viewer visual check and an isolated install/eval.\n\n### Datasets\n- **Primary dataset(s)**: Seeded scenario generator emitting T0-T3 examples.\n- **Source links**: Static Day 2 world data is generated in-repo with\n  `scripts/build_world.py` from simplified public-structure airspace and\n  synthetic obstacle assumptions.\n- **Current split sizes**: 300 train, 60 dev/calibration, and 60 final eval\n  rows. Each split is stratified evenly across T0-T3; final-eval seeds are\n  never used for dial calibration.\n\n### Task\n- **Type**: Multi-turn tool use.\n- **Role**: Remote pilot in command / ops-center operator.\n- **Model decisions**: File or amend plans, query telemetry/weather/airspace,\n  hold, resume, return to launch, land, release payload, abort missions, and\n  override simulator-owned failsafes when justified by the scenario state.\n- **Rubric overview**: Mission value, hard safety violations, reserve and\n  margin policy, procedural compliance, and efficiency. Reward components read\n  only saved `sim_log` snapshots; model prose is inert.\n\n### Quickstart\n\nInstall the public Hub release and run a two-example smoke evaluation:\n\n```bash\nprime --plain env install jarrett/uav-operator@0.1.1\nprime --plain eval run uav-operator -n 2 -r 1 --disable-tui\n```\n\nFor source development, clone the repository and create the complete runtime\nand development environment from the lockfile:\n\n```bash\nuv sync --locked --all-groups\nuv run ruff check .\nuv run pytest -q\n```\n\nRun a small local smoke evaluation:\n\n```bash\nprime --plain eval run uav-operator -n 2 -r 1 --skip-upload --disable-tui\n```\n\nConfigure model, sampling, and saved simulator state:\n\n```bash\nprime --plain eval run uav-operator \\\n  -m poolside/laguna-m.1 \\\n  -n 5 \\\n  -r 1 \\\n  -t 512 \\\n  -T 0.2 \\\n  --skip-upload \\\n  --disable-tui \\\n  --save-results \\\n  --state-columns sim_state,sim_log\n```\n\nNotes:\n- Put task-owned settings under `[env.taskset]` and harness-owned settings\n  under `[env.harness]` in TOML configs.\n- The core sim is event-driven and analytic. It tests supervisory operator\n  judgment, not low-level flight control or 3D collision physics.\n- Inspect or rebuild the static Day 2 world JSON with\n  `uv run python scripts/build_world.py`.\n- Inspect the deterministic scripted rollout artifact at\n  `assets/rollouts/day3_scripted_rollout.json`.\n- Run local baselines with\n  `uv run python scripts/baselines.py --policy both --episodes 20 --tier all`.\n- Run Day 5 scripted calibration with\n  `uv run python scripts/day5_calibration.py`. Build a frontier plot by passing\n  saved `vf-eval` run directories to `scripts/plot_day5_scores.py`.\n- Run each final frontier tier with fixed sampling, for example:\n  `prime --plain eval run uav-operator -m poolside/laguna-m.1 -n 15 -r 2 -t 512 -T 0.2 --save-results --state-columns sim_state,sim_log`.\n  Repeat for `gpt-5-nano` and record the printed run ID/results path; use taskset\n  tier overrides for `T0` through `T3`.\n- For providers with intermittent rollout errors, use\n  `uv run python scripts/run_frontier_calibration.py --model poolside/laguna-m.1`.\n  It archives failed rows beside the run, resumes only missing per-example\n  slots, and retries until every tier has 30 clean rollouts. Pass\n  `--max-attempts N` to cap retries; the default `0` is intentionally unlimited.\n  Individual rollouts still have a 420-second timeout so one hung provider\n  request cannot block the persistent retry loop.\n\n### Offline rollout renderer\n\nThe renderer consumes only a saved JSON object containing frozen-schema\n`sim_log`; it never imports, instantiates, or replays the simulator. The map is\na 2D supervisory-operations display, not a flight-dynamics or 3D collision\nvisualization.\n\n```bash\nuv run python scripts/render.py assets/rollouts/day7/laguna-t3-treasure-island-safe-response.json \\\n  -o out.mp4\n\n# Fully offline vector fallback; no tile cache or network required.\nuv run python scripts/render.py assets/rollouts/day7/laguna-t3-treasure-island-safe-response.json \\\n  -o out.mp4 --no-basemap\n```\n\nEach MP4 is 1920×1080, 30 fps, audio-free H.264/yuv420p. The committed README\nGIF is 960 px wide. Basemap attribution appears in the display and downloaded\ntiles are cached under the ignored `.cache/uav-renderer/` directory.\n\nThe five clips are fixed zero-based rows from Laguna run\n`day6-t3-clean-retry/results.jsonl`; each compact JSON retains source run, row,\nmodel, reward, metrics, and the complete `sim_log`:\n\n| Source row | Evidence | Reward | Clip |\n| ---: | --- | ---: | --- |\n| 23 | Treasure Island: four-event safe response (hero) | 0.865 | [MP4](assets/renders/day7/laguna-t3-treasure-island-safe-response.mp4) |\n| 28 | Pier 39: clean composed-event completion | 0.943 | [MP4](assets/renders/day7/laguna-t3-pier-39-clean-completion.mp4) |\n| 4 | Bay Farm: safe completion at 28.9% battery | 0.700 | [MP4](assets/renders/day7/laguna-t3-bay-farm-low-margin.mp4) |\n| 12 | Treasure Island: repeated geofence/override struggle, then success | 0.044 | [MP4](assets/renders/day7/laguna-t3-treasure-island-geofence-struggle.mp4) |\n| 20 | Richmond Channel: battery-depletion aircraft loss | -4.300 | [MP4](assets/renders/day7/laguna-t3-richmond-battery-loss.mp4) |\n\n[View the five-episode trigger/decision/outcome contact sheet](assets/renders/day7/contact-sheet.png).\n\nThe hero's pop-up restriction activates while the aircraft is still on the\nground during mission intake; it is not presented as a mid-flight TFR. A true\nmid-flight TFR clip and a same-seed before/after-training pair remain deferred\nuntil suitable saved evidence exists.\n\n### Day 6 post-fix calibration\n\n![GPT-4.1-nano and Laguna reward by curriculum tier](assets/evals/day6_tier_scores.png)\n\nThe calibration compares GPT-4.1-nano and Laguna on the same dev split with\nfixed sampling, 30 rollouts per tier, and no provider errors. Error bars are\n95% normal confidence intervals over rollout rewards; `n` is annotated on\nevery point.\n\n| Model | Tier | Mean reward | 95% CI | Missions completed | Hard safety | Run ID |\n| --- | --- | ---: | ---: | ---: | ---: | --- |\n| GPT-4.1-nano | T0 | 0.884 | [0.847, 0.921] | 30/30 | 0 | `cdbac883` |\n| GPT-4.1-nano | T1 | -0.055 | [-0.451, 0.341] | 6/30 | 1 | `d43ac56f` |\n| GPT-4.1-nano | T2 | -0.140 | [-0.158, -0.122] | 0/30 | 0 | `99c64cc4` |\n| GPT-4.1-nano | T3 | -0.196 | [-0.229, -0.162] | 0/30 | 0 | `bde540dd` |\n| Laguna | T0 | 0.998 | [0.996, 1.000] | 30/30 | 0 | `day6-t0-clean-retry` |\n| Laguna | T1 | 0.424 | [0.098, 0.750] | 22/30 | 0 | `day6-t1-clean-retry` |\n| Laguna | T2 | 0.476 | [0.221, 0.731] | 29/30 | 0 | `day6-t2-clean-retry` |\n| Laguna | T3 | 0.095 | [-0.413, 0.603] | 26/30 | 2 | `day6-t3-clean-retry` |\n\nGPT-4.1-nano remains the weak monotonic reference: T0 is near ceiling, T1 is\nmaterially harder, and T2/T3 stay below zero. Laguna is near ceiling on T0 and\nstrong through T2, then drops to `0.095` on T3 with two hard-safety outcomes.\nThe fixed simulator therefore produces the intended frontier separation on\nthe composed-event tier. All 240 plotted rollouts are free of provider errors;\nthe machine-readable results and provenance are in\n[`assets/evals/day6_frontier_calibration.json`](assets/evals/day6_frontier_calibration.json).\n\nDay 6 resolved the Laguna T1 anomaly from the saved rollouts: the T1\n`TFR_POPUP` composition places the restriction over the mission target, and\nLaguna scored 0.93-1.0 on the other three T1 event families but -1.575 on the\nTFR scenarios, burning all 40 turns re-filing conflicting plans instead of\naborting. Geofence holds now flag `mission_target_inside_zone` so that trap is\nlegible. The curve above includes that fix plus Day 6 mid-segment interrupts,\nhover energy, and buffered airspace checks.\n\n### Taskset Config\nPlanned fields:\n\n| Field | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `tier` | string | `mixed_day5` | Scenario tier: `T0`–`T3`, `mixed_day5`, or legacy `mixed_day4`. |\n| `dataset_split` | string | internal | Generated split: 300 train rows, 60 dev rows for scripts, or 60 held eval rows. |\n| `seed` | int | `0` | Base seed for deterministic scenario generation. |\n| `max_examples` | int | `-1` | Limit on dataset size; use `-1` for all generated examples. |\n| `wind_enabled` | bool | `true` | Enable seeded Day 3 wind field and altitude shear. |\n| `gust_front_probability` | float | `0.5` | Probability that an episode includes a gust front. |\n| `system_prompt` | string | ops-manual prompt | Override the operator system prompt (used for red-team/adversarial runs). |\n\n### Harness Config\nPlanned fields:\n\n| Field | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `max_turns` | int | `40` | Maximum operator decision turns per episode. |\n| `sim_time_cap_min` | int | `90` | Maximum simulated episode duration. |\n\n### Metrics\nImplemented rubric metrics:\n\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` | Main scalar reward, computed from simulator state |\n| `mission_value` | Completed mission value after timeliness decay |\n| `hard_safety` | Airspace incursions, aircraft loss, and critical battery outcomes |\n| `margin_policy` | Reserve, override, and minimum-safe-altitude penalties |\n| `procedure` | Alert acknowledgement, conflict filing, and hold-expiry penalties |\n| `efficiency` | Energy and simulated-time cost relative to scenario par |\n","encoding":"utf-8","truncated":false,"total_bytes":10549},"status":null}