{"data":{"kind":"file","path":"README.md","version_id":"vuetwntbdwasju1zdp5mhcin","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":1695,"modified_at":"2026-08-23T22:38:04.220000","content_hash":"748868931e76d8a9b3733862225879f759d60f271604f835bb396ef493bdaf97"},"entries":[],"content":"# escalation-router\n\n### Overview\n- **Environment ID**: `referee-lab/escalation-router`\n- **Short description**: Predict whether a 7B VLM answers the capture-QA question correctly on this clip. This is the cascade's defer decision as an eval.\n- Part of the **referee-lab capture-QA family** (see also `referee-lab/hands-visible`).\n\n### Task\nSingle-turn, multimodal. Gold is measured, not judged: qwen2.5vl:7b was run on every clip (2 repeats, temperature 0.7); easy = all repeats correct, hard = otherwise. Routing gold from real runs of the hands-visible eval.\n\n### Quickstart\n```bash\nprime eval run referee-lab/escalation-router\nprime eval run referee-lab/escalation-router -m <model> -n 10 -r 1\n```\n\n### Metrics\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` / `correct_answer` | 1.0 on exact match with the constructed/measured gold |\n\n### Baselines\n| Model | Accuracy | Scored | Errors |\n| ----- | -------- | ------ | ------ |\n| gemma3:27b | 40.0% | 45 | 0 |\n| qwen2.5vl:7b | 33.3% | 45 | 0 |\n\nTemperature 0, one pass, run on local hardware (GX10).\n**Read:** both models score below the 66.7% constant-'hard' ceiling — predicting a smaller model's per-clip behavior (the cascade's defer decision) defeats current VLMs. The eval measures a metacognition gap.\n Constant-answer\nceiling per validate_family.py output — a useful model must beat it.\n\n\n### Provenance\nAll media derives from DROID (CC BY 4.0) via lerobot/droid_1.0.1 (some items reuse\nthe graded hands-visible corpus). Per-item provenance is machine-readable in\n`assets/dev/manifest.jsonl`; credits in [ATTRIBUTION.md](ATTRIBUTION.md).\nLabels are by construction or from measured model runs — no human grading claimed.\n","encoding":"utf-8","truncated":false,"total_bytes":1695},"status":null}