{"data":{"kind":"file","path":"README.md","version_id":"ku7k0efcq3avkleiwe9yl94w","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":10002,"modified_at":"2026-07-16T15:19:05.382000","content_hash":"38f41b2a376aed081249f5b6387ce74ae82649d8dad15563a0a91f274d8debca"},"entries":[],"content":"# feishu-office-v1\n\nChinese workplace agent tasks in a simulated Feishu (Lark) workspace. The agent acts as a colleague's assistant inside a small fictional company (16 people, 5 departments). It reads group chats, calendars, policy docs, bitable tables and approval queues through MCP tools, then has to leave the workspace in the correct final state. All instructions, chat logs, policies and records are written in natural workplace Chinese.\n\nScoring is fully deterministic. Every task carries typed expected values computed at authoring time, and rewards compare the seeded workspace against the final one. No LLM judge.\n\n## Taskset\n\n140 tasks, 7 families with 20 hand-built variants each, in four tiers. Standard (variants 0 to 4), hard (5 to 9), expert (10 to 14) and frontier (15 to 19). Filter with `taskset.difficulty`.\n\n| Family | The agent has to |\n| --- | --- |\n| `reschedule` | Read a group thread where colleagues negotiate moving a meeting (including rejected proposals), update the right calendar event, and announce the new time in the chat |\n| `schedule` | Find the earliest common free slot for several busy attendees under working-hour and lunch constraints, then create the event |\n| `leave` | Read the leave policy doc and a chat agreement, then file a leave approval with the correct working-day count, dates and handover person |\n| `expense` | Reconstruct trip costs from a chat thread, apply per-category caps from the reimbursement policy, and submit the exact reimbursable total |\n| `approvals` | Process a manager's pending approval queue strictly by a written policy, cross-checking leave dates against the department calendar |\n| `signup` | Tally a group-chat signup thread (including a mind-changer), update only the bitable rows that were actually decided, and report the headcount |\n| `brief` | Aggregate a bitable sales table (signed deals only) and message the director the exact totals and top performer |\n\nThe hard tier layers compound constraints onto the same checks. Cross-week moves with lookalike meetings on the calendar, two-day slot searches for 90-minute meetings, leave-balance rules that silently change the leave type, lost receipts and shared hotel rooms and non-reimbursable entertainment inside the expense math, approval queues that require cross-checking event participants and a department budget table, signup threads with family members, double mind-changers and other-channel noise, and month-over-month aggregation across two tables with test rows to exclude.\n\nThe expert tier was calibrated against measured frontier-model failures. It forces multi-hop chains where the needed fact lives in a different artifact than the instruction. Fallback plans that depend on the organizer's own calendar, three-day slot searches, an invoice registry that overrides misremembered chat amounts plus unregistered items that must be dropped, company holidays that change the day count, GM pre-approvals granted in chat that flip a rejection, duplicate submissions, and refunded deals reported separately from valid revenue.\n\nThe frontier tier multiplies scale times precision while keeping the rule set close to the expert tier: with all-or-nothing scoring, twenty-plus rule applications per task punish a 2% per-item error rate that shorter tasks forgive. A 25-row invoice registry reconciled under two-tier city caps, per-day meal aggregation, shared-room splits, personal-stay exclusions and a business-versus-personal rebooking clause, with colleagues' invoices mixed into the registry. A 14-item approval queue where purchase budget is consumed sequentially in submission order, so one wrong early call cascades, on top of GM pre-approvals, duplicates and calendar cross-checks. Quarter accounting across three monthly tables with backfilled rows re-attributed to their true month and a top performer ranked by deal count, not amount. Signup threads with a cutoff announcement that voids late entries except one the organizer whitelists in DM, plus per-family guest caps and fee arithmetic. Four-day slot searches where one calendar block was cancelled in chat but never deleted, and another constraint exists only in chat. Meeting moves that require cancelling the actor's own colliding sync among identically titled decoys. Leave filings where the annual quota must be derived from hire-date tenure tiers, a leave ledger with rejected, withdrawn and prior-year rows, and a company holiday inside the span. Every planted trap in the builders carries an assert proving the naive reading changes the expected answer.\n\nEach family also verifies restraint. Untargeted events, records and approvals must survive untouched, and several checks catch delete-and-recreate shortcuts or extra submissions.\n\n## Tools\n\nOne task-scoped MCP server (`feishu_*`) with 18 tools over messaging, contacts, calendar (including freebusy), cloud docs, bitable and approval flows. Forms validate required fields and date formats the way the real Open API pushes back, and approvals auto-route to the applicant's manager.\n\n## Rewards and metrics\n\n- `task_success` (reward). 1.0 when every check for the task passes, else 0.0.\n- `checks_passed` (metric). Fraction of the task's checks that passed; per-check booleans are stored in `trace.info[\"feishu\"]` for inspection.\n\n## Quickstart\n\n```bash\nuv run eval feishu_office_v1 -n 5 -r 1\n```\n\nRestrict to one family while iterating\n\n```bash\nuv run eval feishu_office_v1 --taskset.families '[\"expense\"]' -r 3\n```\n\n## Configuration\n\n| Arg | Type | Default | Description |\n| --- | --- | --- | --- |\n| `taskset.families` | list[str] | all 7 | Subset of task families to load |\n| `taskset.difficulty` | str | all | `standard`, `hard`, `expert` or `frontier` |\n\n## Baselines\n\nOne rollout per task. GLM models ran through the default verifiers tool harness; Claude Opus 4.8 ran the same tools and checks through an external CLI harness.\n\n| Tier | GLM-4.6 | GLM-5.2 | Claude Opus 4.8 |\n| --- | --- | --- | --- |\n| standard (35) | 23/35 | 35/35 | not run |\n| hard (35) | not run | 33/35 | 31/35 |\n| expert (35) | not run | 34/35 | 34/35 |\n| frontier (35) | not run | 28/35 | 14/14 sampled |\n\nThe tiers separate model capability cleanly. GLM-4.6 fails a third of the standard tier on relative-date resolution, freebusy intersection, calendar cross-checks and misdirected messages. The flagships diverge on complementary weaknesses. GLM-5.2 loses two-day constraint searches (`schedule` hard 3/5) while Opus solves all of them; Opus loses multi-rule reimbursement arithmetic (`expense` hard 1/5, misapplied hotel and meal caps) while GLM-5.2 solves them. On the expert tier both flagships failed the same task, a reimbursement that hinges on a business-versus-personal rebooking-fee clause.\n\nThe frontier tier was itself calibrated in two rounds. The first draft restated the scoring rubric inside each prompt, and both flagships saturated it (GLM-5.2 34/34, Opus 14/14): a prompt that enumerates the rules does the task decomposition for the model. The shipped version moves every rule into the artifacts (policy docs, announcements, DMs) and adds superseded-policy decoys, after which GLM-5.2 drops to 28/35: `signup` 0/5 (misapplied cutoff, proxy-signup and no-reply rules once they only exist in the organizer's messages), plus one tenure-boundary miss in `leave` and one sequential-budget miss in `approvals`. Opus 4.8 still passes all 14 sampled frontier tasks under a patient external CLI harness, so headroom against thinking-enabled flagships remains a paid-milestone problem, not a claim this tier makes.\n\nAll observed failures are genuine model errors. Every task carries an offline oracle test proving full reward is reachable through the tools alone, and every planted trap carries an authoring-time assert that the naive reading yields a different answer.\n\n## Notes\n\n- The workspace, company and people are fictional, and the simulator is an independent project with no affiliation to Feishu or ByteDance.\n- Weekday words in seeded chats (\"周四\", \"明天\") match the real 2026 calendar of each task's anchor date, so relative-date resolution is part of the challenge.\n- Message checks accept common date and time spellings (7月23日, 07-23, 16:00, 16点) and normalize whitespace and separators before matching numbers.\n\n## Changelog\n\n#### v0.4.1\n\nLegacy `load_environment` bridge so classic `vf-eval` (and the Hub integration test) can drive the same scenarios, tools and checks through the v0 `StatefulToolEnv` API. Free-form `dict` tool parameters are exposed as JSON strings to keep strict schemas.\n\n#### v0.4.0\n\nFrontier tier (35 tasks, 140 total). Scale-times-precision under de-scaffolded prompts: every rule lives in an artifact, none in the prompt. 25-row invoice reconciliation with two-tier city caps and per-day meal aggregation, 14-item approval queues with sequentially consumed purchase budgets, quarter accounting across three tables with backfill re-attribution and count-based ranking, signup cutoffs with DM-whitelisted exceptions and proxy signups, five-day slot searches with near-miss gaps and chat-corrected calendars, conflict-cancelling reschedules among identical titles, and hire-date tenure quotas reconciled against a leave ledger. Superseded-policy decoy docs for the three policy families. Two new check fields (`cancel_event_id`, `fee_total`). Two-round difficulty calibration against GLM-5.2 and Claude Opus 4.8.\n\n#### v0.3.0\n\nExpert tier (35 tasks). Multi-hop artifact chains, three-day slot searches, invoice-registry reconciliation, company-holiday day counts, GM pre-approval and duplicate-submission clauses, refund accounting. Cross-model baselines for GLM-4.6, GLM-5.2 and Claude Opus 4.8.\n\n#### v0.2.0\n\nHard tier (35 tasks) with compound constraints per family, `taskset.difficulty` filter, GLM baselines.\n\n#### v0.1.0\n\nInitial release. 7 families, 35 tasks, deterministic state-diff scoring, offline oracle tests proving every task is solvable to full reward through the tools alone.\n","encoding":"utf-8","truncated":false,"total_bytes":10002},"status":null}