{"data":{"kind":"file","path":"README.md","version_id":"uww816eyxqf3dp1mlv5we8ta","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":1957,"modified_at":"2026-08-24T03:09:11.563000","content_hash":"17952daae09732595959174a284dcbffab7f3f6ca94165d19eaa9a82ed2e5c87"},"entries":[],"content":"# effective-fps\n\n### Overview\n- **Environment ID**: `referee-lab/effective-fps`\n- **Short description**: Is the temporal sampling genuine, or upsampled from a lower true capture rate? Detects frame-duplication fraud (footage sold at 30fps that was captured at 10-15fps).\n- Part of the **referee-lab capture-QA family** (see also `referee-lab/hands-visible`).\n\n### Task\nSingle-turn, multimodal. Gold by construction: genuine items are true consecutive frames; upsampled items repeat each real frame 2-3x, exactly what naive rate upsampling produces.\n\n### Quickstart\n```bash\nprime eval run referee-lab/effective-fps\nprime eval run referee-lab/effective-fps -m <model> -n 10 -r 1\n```\n\n### Metrics\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` / `correct_answer` | 1.0 on exact match with the constructed/measured gold |\n\n### Baselines\n| Model | Accuracy | Scored | Errors |\n| ----- | -------- | ------ | ------ |\n| gemma3:27b | 55.0% | 40 | 0 |\n| qwen2.5vl:7b | 50.0% | 40 | 0 |\n\n| EgoSieve-S `duplicate_frames` head (our curation model, own threshold) | 50.0% | 40 | 0 |\n\nTemperature 0, one pass, run on local hardware (GX10). EgoSieve-S — our own\ncuration model — scores exactly chance here (0/20 on upsampled items): its\ntraining corruptions do not transfer to real rate-upsampling. Grading our own\nmodels first is the house rule; a v0.2 trained against this eval is the obvious\nnext step.\n**Read:** barely above the 50% coin-flip ceiling — frame-duplication detection is genuinely unsolved for current VLMs. Wide open.\n Constant-answer\nceiling per validate_family.py output — a useful model must beat it.\n\n\n### Provenance\nAll media derives from DROID (CC BY 4.0) via lerobot/droid_1.0.1 (some items reuse\nthe graded hands-visible corpus). Per-item provenance is machine-readable in\n`assets/dev/manifest.jsonl`; credits in [ATTRIBUTION.md](ATTRIBUTION.md).\nLabels are by construction or from measured model runs — no human grading claimed.\n","encoding":"utf-8","truncated":false,"total_bytes":1957},"status":null}