{"data":{"kind":"file","path":"README.md","version_id":"lox8c94h8js7cf7fgzqax6wl","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":1998,"modified_at":"2025-10-17T03:27:07.559000","content_hash":"f7fbd7f9b349f18cb633e4746cbe7fd1baf9cc6ce7bebb96854ab375ad9a414b"},"entries":[],"content":"# OCRBench\n\nOCRBench is a benchmark for evaluating the optical character recognition capabilities of Large Multimodal Models, featuring multiple versions of datasets to assess text extraction and understanding from images.\n\nPaper refs: [Version 1](https://arxiv.org/abs/2305.07895), [Version 2](https://arxiv.org/abs/2501.00321)\n\n### Overview\n- **Environment ID**: `OCRBench`\n- **Short description**: A benchmark for evaluating optical character recognition capabilities of Large Multimodal Models\n- **Tags**: `multimodal`, `ocr`, `text-extraction`, `vision`\n\n### Datasets\n- **Primary dataset(s)**: OCRBench v1 and v2 datasets\n- **Source links**: [Version 1](https://huggingface.co/datasets/echo840/OCRBench), [Version 2](https://huggingface.co/datasets/lmms-lab/OCRBench-v2)\n- **Split sizes**: Version 1 (`1k`), Version 2 (`10k`)\n\n### Task\n- **Type**: `single-turn`\n- **Parser**: Custom parser for extracting answers wrapped in `**`\n- **Rubric overview**: Exact match accuracy on OCR answers, supports filtering by dataset and question type\n\n### Quickstart\nRun an evaluation with default settings:\n\n```bash\nuv run vf-eval OCRBench\n```\n\nConfigure model and sampling:\n\n```bash\nuv run vf-eval OCRBench -k <API_KEY> -b <BASE_URL> \\\n  -m 'mistralai/mistral-small-3.2-24b-instruct' \\\n  -n 5 -r 1 -s -a '{\"num_samples\": 5, \"version\": \"1,2\"}'\n```\n\nNotes:\n- Use `-a` / `--env-args` to pass environment-specific configuration as a JSON object.\n\n### Environment Arguments\n\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `num_samples` | int | `-1` | Limit on dataset size (use -1 for all) |\n| `version` | str | `'1,2'` | OCRBench dataset version(s) to use |\n| `exclude_ds` | List[str] | `[]` | List of datasets to exclude |\n| `exclude_types` | List[str] | `[]` | List of question types to exclude |\n\n### Metrics\n\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` | Main scalar reward (1 for correct answer, 0 for incorrect) |\n| `accuracy` | Exact match on target answer |\n\n","encoding":"utf-8","truncated":false,"total_bytes":1998},"status":null}