{"data":{"kind":"file","path":"README.md","version_id":"scxybdhybywmdbmetopbr2kl","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6468,"modified_at":"2026-08-24T19:32:36.901000","content_hash":"45a91c92faaa1b1d530d8c7a5e1fb5e3be81edc7a7ae45caf62302d382490c86"},"entries":[],"content":"# hands-visible\n\n### Overview\n- **Environment ID**: `referee-lab/hands-visible`\n- **Short description**: Is at least one human hand or robot gripper clearly visible\n  anywhere in the clip? Capture-QA eval for robot-training video, scored against a\n  graded answer sheet.\n- **Tags**: eval, multimodal, robotics, vision, capture-qa\n\n### Task\n- **Type**: single-turn, multimodal (12 frames per clip as images)\n- **Rubric overview**: `correct_answer` — exact match of the reply's final yes/no\n  against the answer sheet. Grading rules for the answer sheet are in\n  [TASK_CARD.md](TASK_CARD.md); allowed data sources in [DATASETS.md](DATASETS.md).\n\n### Status\nReal-footage corpus in progress. The bundled split mixes real robot clips (DROID\nCC BY 4.0, aloha sim MIT), real aerial-wildlife negatives (KABR CC0), and synthetic\ncontrol scenes. Labels are **machine-graded provisional** (every clip individually\nreviewed from contact sheets; grading record in the repo) until the two-grader\nhuman answer sheet from the pilot lands. Treat leaderboard numbers as provisional\nuntil that note is gone.\n\n### Corpus scope (v0.3)\n the wildlife negatives all come\nfrom one KABR drone session, and positives are all robot workspaces — scene composition is\npart of the measured task in v0.3. The v0.4 roadmap adds robot-scene negatives\nand a human-egocentric tier (positives and negatives from the same footage\ndomain). Per-clip provenance is machine-readable in the manifest\n(`source`, `source_license`); full credits in [ATTRIBUTION.md](ATTRIBUTION.md).\n\n### Quickstart\n```bash\nprime eval run referee-lab/hands-visible\nprime eval run referee-lab/hands-visible -m <model> -n 20 -r 1 \\\n  -a '{\"data_dir\": \"my_corpus\", \"max_examples\": 50}'\n```\n\n### Environment Arguments\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `data_dir` | str | bundled split | Path to a corpus: `manifest.jsonl` + frame JPEGs |\n| `max_examples` | int | `-1` | Limit on dataset size (-1 for all) |\n| `min_difficulty` / `max_difficulty` | float | unset | Band clips by measured difficulty (1 − fleet pass-rate). Clips without a `difficulty` field (all bring-your-own corpora until you run `run_fleet.py`) are excluded when banding; an empty band raises. |\n\n### Metrics\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` / `correct_answer` | 1.0 when the reply's final yes/no/unclear matches the answer sheet |\n| `abstained` (weight 0) | fraction of replies answering \"unclear\" |\n\n### Baselines (bundled 45-clip split, corpus r2, abstain-channel prompt)\nRepeated sampling at temperature 0.7 (2 repeats per clip); accuracy averaged\nover repeats. A constant \"yes\" answers 68.9% — treat that as the floor.\n\n| Model | Accuracy | Yes-recall | No-recall |\n| ----- | -------- | ---------- | --------- |\n| gemma3:27b | 97.8% | 100% | 92.9% |\n| qwen2.5vl:72b | 83.3% | 77.4% | 96.4% |\n| qwen2.5vl:32b | 74.4% | 67.7% | 89.3% |\n| qwen2.5vl:7b | 45.6% | 25.8% | 89.3% |\n| qwen2.5vl:3b | 33.3% | 17.7% | 67.9% |\n| MediaPipe HandLandmarker (detector, not a VLM) | 40.0% | 12.9% | 100% |\n| **referee-lab student** (MobileNetV3-small, 1.5M params, distilled from gemma3) | **97.8%** | 96.8% | 100% |\n| referee-lab student, cross-video split (no training frame within 60s of a test clip) | 95.6% | 93.5% | 100% |\n| EgoSieve-S (our HoloAssist-trained curation model, own threshold) | 40.0% | 12.9% | 100% |\n\nThe student ships in two splits. The standard row trained on frames from\ndifferent clips but the same source streams as this corpus. The cross-video row\nremoves every training frame within 15 clip indices (>=60 seconds) of any test\nclip on its DROID stream; KABR training data comes from disjoint drone sessions\nby construction. Under that separation the student scores 100% on real footage\n(41/41) — its only misses are the two synthetic drawn-hand controls, which sit\noutside its training distribution on purpose. Training: `edge/train_student.py\n--cross-video`; per-clip results in `student_row_crossvideo.json`. EgoSieve-S is our own\ncuration model, trained on HoloAssist human video — its row here measures\ndomain transfer to robot footage, and it shows the same gripper-blindness as\nMediaPipe. The league grades its own models by the same rules as everyone\nelse's; that is the point of it.\n\nMediaPipe's row measures domain transfer, not general skill: it is perfect on\nnegatives (never hallucinates a hand) and blind to robot grippers (2/29 on the\nrobot tier) — a human-hand detector cannot referee gripper footage at any\ncompute tier. Its human-tier row lands when the egocentric corpus does.\n\nThe eval discriminates hard: clean scaling within the qwen2.5vl family, only the\ntop two rows beat the constant-yes floor, and the abstain channel punishes weak\nmodels that escape-hatch to \"unclear\" on readable clips. Numbers are not\ncomparable to pre-0.3.0 rows (the prompt changed). All rows ran on local\nhardware; no hosted APIs involved.\n\n### Validation\nEvery push runs `python validate.py`: manifest integrity (no dupes, no orphan or\nmissing frames), a decode gate over every frame, a no-op check (garbage\nsubmissions score ~0), and a gold check (the answer sheet scores 100%). The\nmanifest also carries per-clip `difficulty` (1 − fleet pass-rate over the\nbaseline runs) and `unstable` (the clip flip-flopped within a model across\nrepeated sampling) — band with `min_difficulty` / `max_difficulty`.\n\n### Bring your own dataset\nThe environment runs on any corpus you have rights to — including research-only or\nnon-commercial datasets you may not redistribute (see DATASETS.md for verdicts).\n\n1. Put clips in a folder, write `labels.csv` (header `filename,answer`, yes/no;\n   `grade.py` can also record `unclear` for unreadable media).\n2. `python make_manifest.py my_videos/ --labels labels.csv --out my_corpus`\n3. `prime eval run referee-lab/hands-visible -a '{\"data_dir\": \"my_corpus\"}'`\n\nAlready have frames? Write `manifest.jsonl` directly: one line per clip,\n`{\"clip_id\", \"frames\": [...], \"answer\": \"yes\"|\"no\"}`, paths relative to the folder.\n\n### Grading kit\n- `TASK_CARD.md` — the yes/no rules, written so two graders agree.\n- `grade.py run <dir> --grader <name>` — contact-sheet grading, resume-safe.\n- `grade.py merge <dir> --graders a,b` — agreement + Cohen's kappa + pilot gate\n  (≥95%, κ≥0.8), writes `labels.csv`.\n- `make_manifest.py` — turns clips + labels into a corpus.\n- `python test_env.py` — offline smoke test, no API key needed.\n","encoding":"utf-8","truncated":false,"total_bytes":6468},"status":null}