{"data":{"kind":"file","path":"README.md","version_id":"zwiqqeivnhkdbi7bpb73eyz1","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":11261,"modified_at":"2026-07-31T18:46:24.874000","content_hash":"87a22c84aa63d6fce9c5058bae9196368ab109b8283e592b8af65776ca8553ee"},"entries":[],"content":"# enterprise-vendor-eval\n\nAn RL environment for **enterprise SaaS vendor evaluation** — the procurement/security-review job as it actually exists inside large deskless-workforce organizations (cruise lines, hotel groups, airlines' ground handlers, retail chains, home-health providers, logistics operators).\n\nThe agent is a procurement analyst. Each episode drops it into a vendor deal with a document store — an internal **requirements matrix**, the vendor's **security questionnaire response**, a **SOC 2 excerpt**, an **internal IT/procurement thread**, sometimes a pricing addendum, architecture note, admin guide or trust-center page — plus an internal IT channel it can ask. It must investigate and submit a **go / conditional-go / no-go** recommendation flagging **every unmet requirement by its requirement ID**.\n\n**Fully deterministic verifier. No LLM judge.** All companies, vendors, documents and people are fictional.\n\n### Overview\n- **Environment ID**: `enterprise-vendor-eval`\n- **Short description**: Multi-turn tool-use vendor due-diligence: audit hedged, incomplete, sometimes self-contradicting vendor documents against a requirements matrix; flag every real gap, resist every planted distractor.\n- **Tags**: `procurement`, `enterprise`, `security-review`, `vendor-evaluation`, `tool-use`, `multi-turn`, `train`, `eval`\n\n### What makes it hard (and realistic)\n\nThe difficulty is in the *documents*, not the mechanics:\n\n- **Vendor prose reads like vendor prose.** Hedges (\"industry-standard practices\"), roadmap answers presented as capabilities, refusals dressed as policy, marketing claims that the DPA quietly contradicts.\n- **Gaps by absence.** Medium scenarios plant gaps as *silence* — the questionnaire simply never addresses a requirement, and noticing what's missing is the task.\n- **Cross-document contradictions.** Hard scenarios hide the truth in reconciliation: the questionnaire says \"fully offline\", the admin guide caps offline attestations at 7 days; the trust center says \"never trains on your data\", DPA Annex 5 default-enrolls you in a Model Improvement Program; \"EU hosting\" is true for storage while 7.2% of peak traffic is *processed* in US-Virginia.\n- **Distractors.** Every scenario plants 1–3 requirements that *look* unmet but are satisfied somewhere in the file (buried in an appendix, evidenced only in the SOC 2, or debunked stale rumors). Flagging them costs points.\n- **Zero-gap scenarios.** Two scenarios have *no* gaps — the correct answer is \"go\" with an empty gap list. Anti-hallucination pressure is built into the score.\n- **Priority & remediation logic.** MUST vs SHOULD gaps and documented committed remediation dates determine the correct decision via a fixed rule, so the decision itself is derivable — and gradable — from the file.\n\n### Task\n- **Type**: multi-turn tool use\n- **Tools**:\n  - `search_docs(query)` — deterministic keyword search over the document store with snippets\n  - `read_doc(doc_id)` — full document text\n  - `ask_it_channel(question)` — canned, keyword-routed replies from internal colleagues (deterministic)\n  - `submit_recommendation(decision, gap_ids, rationale)` — ends the episode\n- **Episode end**: the episode stops the moment `submit_recommendation` is called (or on max_turns / no tool calls).\n\n### Verifier (deterministic)\n\nGaps are flagged by **requirement ID** (`REQ-SSO`, `REQ-SUBPROC`, …) exactly as defined in each scenario's requirements matrix — scoring is exact set-matching against authored ground truth, mirroring how real procurement teams work off a requirement matrix.\n\n| Reward | Weight | Logic |\n| ------ | ------ | ----- |\n| `reward_gaps` | 0.7 | `max(0, (hits − 0.5·false_flags) / planted)`; zero-gap scenarios: `max(0, 1 − 0.5·false_flags)` |\n| `reward_decision` | 0.2 | exact match on go / conditional-go / no-go (derived by a fixed, prompt-stated rule) |\n| `reward_process` | 0.1 | 1.0 only if every core document was read before submitting |\n\nDiagnostics (weight 0): `metric_submitted`, `metric_false_flags`, plus verifiers' built-in tool-call and turn metrics.\n\n### Dataset\n\n20 hand-written scenarios (6 easy / 9 medium / 5 hard), each with 4–6 documents (~700–1,200 words of document text per scenario), 6–9 requirements from a fixed 24-item taxonomy, 0–4 planted gaps, 1–3 distractors, and a keyword-routed IT channel. Six fictional buyer organizations and nine fictional vendors recur across scenarios with **consistent histories** (a vendor that refused to publish a subprocessor list in an early-dated scenario has a complete register in a later one — and one hard scenario tests whether the agent evaluates the current file instead of pattern-matching on the vendor's stale reputation).\n\nScenario JSONs live in `enterprise_vendor_eval/scenarios/` and ship inside the wheel. `scripts/validate_scenarios.py` enforces the schema: taxonomy membership, decision-rule consistency, planted/distractor disjointness, no internal REQ-IDs leaking into vendor-authored documents, and no giveaway phrasing.\n\n**The colleagues must not adjudicate.** The first version of this environment scored 0.99 with a frontier model because the IT thread and `ask_it_channel` replies stated verdicts outright — *\"MUST unmet, flag it\"*, *\"Met — do not flag this one\"*. An agent could skip the vendor documents entirely and ask a colleague for the answer key. A second pass removed a subtler version of the same leak: colleagues citing the exact section holding each trap (*\"read their Q5 and check the dates\"*), which did the hard part of the cross-document reasoning for the reader.\n\nColleagues now supply what a colleague plausibly has — business pressure, pilot observations, and facts absent from the documents (\"I asked their AE for a date in the order form and he declined\") — and never a verdict or a pointer to where a problem lives. This is enforced mechanically by `VERDICT_LEAK_PATTERNS` in the validator plus a test, so scenario edits cannot quietly reintroduce the shortcut.\n\n### Quickstart\n\nRun an evaluation with default settings (defaults to Prime Inference; set `PRIME_API_KEY`):\n\n```bash\nprime env install enterprise-vendor-eval\nprime eval run enterprise-vendor-eval -m openai/gpt-4.1-mini -n 20 -r 3\n```\n\nOr with the verifiers CLI against any OpenAI-compatible endpoint:\n\n```bash\nuv run vf-eval enterprise-vendor-eval -n 20 -r 3 -m <model> -b <base_url> -k <API_KEY_VAR>\n```\n\nFrom a source checkout:\n\n```bash\nuv venv && uv pip install -e . --group dev\nuv run pytest tests            # 23 deterministic tests\nuv run python scripts/scripted_rollouts.py   # no-LLM harness verification\nuv run python scripts/validate_scenarios.py  # scenario schema + leakage gate\n```\n\n**Windows note.** Two things bite on Windows and neither is specific to this environment: `verifiers` needs `tornado` for pyzmq's Proactor event loop (`uv pip install tornado`), and the ZMQ env-server subprocess does not become healthy — pass `--disable-env-server` to run the environment in-process. Both eval commands above work once those are handled.\n\n### Scripted baselines (no LLM — harness verification)\n\n| Agent | Behavior | Mean reward |\n| ----- | -------- | ----------- |\n| `oracle` | reads core docs, submits ground truth | **1.000** |\n| `paranoid` | flags planted gaps **and** all distractors | 0.710 |\n| `lazy` | reads only the requirements doc, submits \"go\" | 0.090 |\n\nThe ordering (lazy < paranoid < oracle) is asserted by the script; the oracle's 1.000 across all 20 scenarios verifies the reward path end-to-end.\n\n### Model results\n\nAll 20 scenarios, temperature 0.3, via Prime Inference. DeepSeek and Llama at 3 rollouts/example (n=60), Mistral at 2 (n=40).\n\n| Model | Reward | `reward_gaps` | `reward_decision` | `reward_process` | False flags/rollout |\n| ----- | -----: | ------------: | ----------------: | ---------------: | ------------------: |\n| DeepSeek v3.2 | **1.000** | 1.000 | 1.000 | 1.000 | 0.00 |\n| Mistral Small 2603 | 0.920 | 0.900 | 0.950 | 1.000 | 0.05 |\n| Llama 3.3 70B | 0.485 | 0.436 | 0.667 | 0.467 | 0.77 |\n\n**Read this honestly: the environment is saturated at the frontier.** DeepSeek v3.2 solves every scenario, including all five hard ones, with zero false positives. The environment discriminates sharply *below* the frontier — a ~52-point spread down to Llama 3.3 70B — and the weaker models fail in the ways the rubric was built to catch: Llama hallucinates gaps that were never planted (0.77/rollout), submits before reading the document set (`reward_process` 0.47), and averages 5.7 turns against DeepSeek's 11.6.\n\nThat makes this a useful **RL training** environment today (dense, deterministic reward with a real difficulty gradient) and a useful **eval** for mid-tier models — but not a frontier eval. See *Known limitations* for what v0.2 needs.\n\nReproduce:\n\n```bash\nuv run vf-eval enterprise-vendor-eval -p prime -m deepseek/deepseek-v3.2 -n 20 -r 3 -T 0.3\n```\n\n### Known limitations\n\n- **Frontier saturation.** For grading to stay deterministic, every gap must be *findable* from the documents — and a frontier model that reads all four documents carefully finds them. Making gaps unfindable would make ground truth arguable, which is a worse failure. The honest lever is raising *recall burden and false-positive pressure*, not hiding evidence.\n- **v0.2 plan.** Expand each scenario from ~7 to ~14 requirements, where the added ones are satisfied but evidenced somewhere non-obvious (an appendix, a control in the audit report, a different document than the reader expects). This multiplies the opportunities to both miss a real gap and over-flag a satisfied one, without introducing ambiguity. Llama's 0.77 false-flag rate shows that pressure already bites when a model is uncertain.\n- **Reward weighting.** `reward_gaps` at 0.7 dominates; a model can score 0.3 by submitting a correct decision with an empty gap list on a \"go\" scenario. The zero-gap scenarios (s014, s019) exist partly to keep that from being a free ride, but the weighting is worth revisiting alongside v0.2.\n\n### Taskset config\n\n| Field | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `max_turns` | int | `20` | Max assistant turns per episode (focused reviews finish in 6–10) |\n\n### Design notes\n\n- **Determinism trick**: requirement IDs turn a fuzzy judgment task (\"did the model spot the SOC 2 scope mismatch?\") into exact set-matching, with zero judge cost — while staying faithful to how procurement actually documents requirement compliance.\n- **The content is the moat**: the scenario documents are written the way real vendor files read — hedged, incomplete, occasionally self-contradicting — including SOC 2 auditor-language details (scope notes, exceptions with management responses, Type I vs Type II, bridge-letter dynamics), DPA notification-window clauses, seat-pricing arithmetic against seasonal workforce shapes, and works-council/regulatory retention constraints pulling in *both* directions (one scenario requires long retention, several forbid it — pattern-matching \"long retention = bad\" loses points).\n- All companies, vendors, auditors and people are fictional; any resemblance to real entities is coincidental.\n\n### License\n\nMIT\n","encoding":"utf-8","truncated":false,"total_bytes":11261},"status":null}