{"data":{"kind":"file","path":"README.md","version_id":"b08cpjwp1rcmjtwpvic8u6w1","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3140,"modified_at":"2026-10-25T08:11:59.574000","content_hash":"3ee57e01c4524ef0bcf673b89b4c3cba9f7bf5762aa05ba71068e0d1fecfb56e"},"entries":[],"content":"# CAPTCHA Visual Lab v2 — authorized synthetic visual-agent benchmark\n\nA reproducible, procedurally generated **visual reasoning** environment for Prime Intellect Verifiers v1. All puzzles are synthetic and rendered locally. This project does not access, automate, or bypass CAPTCHA services on real sites.\n\n## What changed from v1\n\n- **Image-grounded** puzzles rendered to PNG instead of text-only descriptions.\n- **Five families:** glyph reading, counting, attribute-based selection, scene-change comparison, and navigation.\n- **Four difficulty tiers** and separate `train`, `validation`, `test`, `ood` seeds.\n- Deterministic generation with per-scene fingerprints, plus a data collision audit.\n- Exact-match reward; selection F1 and output-format compliance are *diagnostic* metrics only.\n- Honest offline baselines (blind random, constant, and privileged oracle). Oracle is a generator integrity test, **not** a model result.\n- Verifiers v1 adapter supporting multimodal image messages. Requires a compatible vision model and SDK for end-to-end evaluation.\n\n## Install and try\n\n```bash\npython -m pip install -e .\ncv-lab preview --index 7 --out outputs/example.png\ncv-lab audit --n 250 --out outputs/audit.json\ncv-lab baseline --n 500 --out outputs/baselines.json\npython -m pip install -e '.[test]'\npytest -q\n```\n\nPrime integration (`verifiers>=0.4.0,<0.5`) is installed by default. Verify discovery with `vf-eval captcha-visual-lab-v2 -n 1 --dry-run --no-push`. A live evaluation requires a compatible vision model. **No claim of live model performance or bounty eligibility is made.**\n\n## External model evaluation\n\nProvide JSONL records in the format:\n\n```json\n{\"index\":0,\"split\":\"test\",\"response\":\"ANSWER: 2,4\"}\n```\n\nThen `cv-lab score --input predictions.jsonl --out outputs/model_results.json`.\nThe scoring command reconstructs gold locally. Do not ship hidden test seeds or answer-bearing scene objects to the model.\n\n## Benchmark protocol for a credible bounty submission\n\n1. Freeze generator revision and test configuration, record commit hash and seed commitments.\n2. Run at least 3 vision-capable models across multiple random seeds and report per-family, per-difficulty accuracy with Wilson intervals.\n3. Include a text-only/blind baseline and image-ablation test to verify actual image dependence.\n4. Evaluate distribution shift on the `ood` split, check prompt leakage, duplicate images, malformed outputs, and over-selection behavior.\n5. Record runtime, token cost, and failure examples. Keep holdout seeds private until scoring is complete.\n6. Demonstrate a real Verifiers v1 taskset run and check the current Environment Hub submission/bounty requirements before applying.\n\n## Limits\n\nThis is a research-ready **candidate**, not a proven competitive bounty entry. It currently has no measured vision-model results, no interactive browser tool loop, and no evidence of acceptance by Prime Intellect. The scene generator and local gold answers are inspectable in source, so genuinely blind evaluation needs separately held-out seeds and controlled execution.\n\nMIT licensed. Use only on owned synthetic benchmark assets.\n","encoding":"utf-8","truncated":false,"total_bytes":3140},"status":null}