{"data":{"kind":"file","path":"README.md","version_id":"sm0stzg7z3ajc85hy3bv35cu","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3441,"modified_at":"2026-10-13T01:12:14.768000","content_hash":"f9daaef3e270b3a4a6e19f4b8506838d634ec6bfebf09eda06a79e904ae9bf1c"},"entries":[],"content":"# Spreadsheet Agent Benchmark — V0.1.0 (Spreadsheet Navigation)\n\nA tool-use Verifiers environment for navigating synthetic workbooks. The default `mode=\"tools\"` prompt contains only a question: no CSV rows or answer. The agent calls `list_sheets`, `get_sheet_schema`, `read_range`, `find_rows`, or `calculate` and returns exactly `{\"answer\": 123.45}` (or a string for rankings). Tools operate on a separate workbook for each rollout. No arbitrary Python execution is exposed. A backward-compatible `mode=\"text\"` retains the V0.0.1 four-family reasoning benchmark.\n\n## Tasks and data\n\nSixteen task families: quarter revenue/units, return rate, largest order, category revenue/margin/returns, top product, region revenue/margin, store profit, top region, bottom store, category-region revenue, top category profit and Q1→Q3 growth. `difficulty=\"easy\"` selects four Orders-only families; medium selects eight one-/two-lookup families; hard/expert select four cross-sheet families. Hard/expert add stale Old_Orders, Summary, SalesTargets and Notes; expert adds PivotCache. Expert currently has the same core questions as hard and one extra decoy; row count is unchanged. Numeric answers use unrounded Decimal calculations and half-up rounding at the end; returned orders are excluded unless the question is about return rate.\n\nSeeded generation produces 60 current Orders (20 each in Q1–Q3), Products and Stores. `split=\"train\"` defaults to 64 tasks from a disjoint seed range; `split=\"eval\"` defaults to 32. `examples` may be 1–256. Train and eval generators and scorers are public: this is not a secret hosted holdout.\n\nThe answer is the sole reward (binary exact categorical or numeric within ±0.005). Diagnostic zero-weight metrics are `tool_efficiency` (1/(1+tool calls)), `retrieval_precision` (fraction of retrieved rows from sheets relevant to that family) and `invalid_call_rate`. These are simple proxies, not optimal-path scores. Per-rollout state also tracks `tool_calls_count`, `rows_read`, `relevant_rows_read`, `invalid_calls`. Tool reads are paged at 100 rows per call. `max_turns` defaults to 12, counts model responses and can be set from 2–30.\n\n## Run\n\n```bash\nuv venv --python 3.12 .venv\nuv pip install --python .venv/bin/python -e .\n.venv/bin/python -m unittest discover -s tests -v\n.venv/bin/python tests/rollout_smoke.py\nuv build --wheel\n```\n\n```python\nfrom spreadsheet_agent_benchmark import load_environment\nnavigation = load_environment(split=\"eval\", examples=16, difficulty=\"hard\", mode=\"tools\", max_turns=12)\nlegacy = load_environment(mode=\"text\", examples=8)\n```\n\n`verifiers==0.1.8` is pinned because the v0 ToolEnv/Rubric loader is incompatible with later v1-only releases. The deterministic rollout smoke test uses a scripted client to verify actual tool dispatch and scoring; it is not a real model benchmark or a leaderboard.\n\n## Limitations and future work\n\nThis is V0.1 navigation, not V0.2 full spreadsheet agent or V0.3 editing. Sheets are virtual tables, not .xlsx files; no real file writes, arbitrary Python, OCR, formulas, date-normalization/dirty data, evidence grading, private generator, 10k-row scale, or model leaderboard. The `mode=\"python\"` request fails explicitly instead of silently pretending it is supported. For trustworthy comparisons, report model, budget, score by family and tool metrics; do not treat synthetic diagnostic scores as measured model performance.\n\nLicense: MIT.\n","encoding":"utf-8","truncated":false,"total_bytes":3441},"status":null}