{"data":{"kind":"file","path":"README.md","version_id":"g8rcml950qauhdau0nsj26sm","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2036,"modified_at":"2026-09-13T04:06:06.935000","content_hash":"a3600e89034df5e0217ff64b98a3b7e23d0ec2a9c4b5dec8112e0ddc111e5b90"},"entries":[],"content":"# charxiv\n\nCharXiv chart-understanding questions over arXiv figures ([paper](https://arxiv.org/abs/2406.18521), NeurIPS 2024). Each task shows one chart plus a question; an LLM judge extracts the final answer and scores it 0/1 against the ground truth using the upstream grading rubrics.\n\n## Taskset\n\n- **Source:** [`princeton-nlp/CharXiv`](https://huggingface.co/datasets/princeton-nlp/CharXiv), `validation` split (1000 figures) — the only split with public answers (`test` answers are withheld for the leaderboard)\n- **Modes (`--env.taskset.mode`):**\n  - `reasoning` (default): each figure's free-form reasoning question — 1000 tasks\n  - `descriptive`: the four templated perception questions per figure (19 templates: titles, axis labels, ticks, legends, trends, layout) — 4000 tasks\n- **Prompts:** upstream response instructions verbatim (vendored in [`constants.py`](charxiv/constants.py) from [`src/constants.py`](https://github.com/princeton-nlp/CharXiv/blob/main/src/constants.py)); user message is the question text followed by the chart image, as in the upstream inference scripts\n- **Scoring:** upstream grading prompts with structured-output JSON verdicts (`extracted_answer`, binary `score`); reasoning rubrics are keyed by answer type (text-in-chart, text-general, number-in-chart, number-exact), descriptive rubrics by question template\n- **Judge:** `--env.taskset.task.judge.model`, default `qwen/qwen3.6-35b-a3b` (temperature 0, output capped at 8192 tokens so a judge stuck in a reasoning loop errors cheaply instead of mis-scoring; re-score such rollouts with `--resume`). For harder grading, `google/gemini-3.5-flash` is a good switch.\n\n## Deviations from upstream\n\n- The judge model is configurable (default `qwen/qwen3.6-35b-a3b`); upstream pins `gpt-4o-2024-05-13`.\n- Descriptive responses are graded one per judge call instead of upstream's batches of five same-template responses (`NUM_TRIPLETS`=1 in the same grading prompt; batching was a cost optimization).\n\n## Changelog\n\n- 2026-07-19: Initial port.\n","encoding":"utf-8","truncated":false,"total_bytes":2036},"status":null}