{"data":{"kind":"file","path":"README.md","version_id":"dwqsiappgf78x4f5qygn67e2","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":10250,"modified_at":"2026-09-02T17:39:44.625000","content_hash":"e33cdaa4f435da28c94540da1a7163c03f7f1c7de961be4ff6f8de02cf57c6da"},"entries":[],"content":"# semantic-cache\n\nA `verifiers`-spec **single-turn RL environment** for **cache-safety judging**: given a new query, a cached query, the cached answer, and the embedding cosine between the two queries, the model must decide whether serving the cached answer for the new query is safe (`equivalent`) or not (`not_equivalent`).\n\nEnvironment: **`gutchapa/semantic-cache`** · version **0.1.5** · license **MIT** · published as a **public** Hub environment.\n\n## Why this task\n\nThe production token-saver chain (`client → rtk_proxy → cache_proxy → api`) serves cached responses with an exact-match LRU cache first, then a **semantic cache** that replays a cached answer when `cosine(new_query, cached_query) ≥ 0.95` (embedding model `qwen3-embedding-8b` via Fireworks). That chain saves **81.2%** of tokens (≈1.23M) in a real agentic-coding workload.\n\nThe fixed threshold has two known failure bands, and this environment trains a judge to replace it:\n\n- **Wrong serves** — queries that are semantically *different* but word-overlap enough to sit at/above 0.95 (entity swaps, near-misses, homographs). Serving the cached answer is actively harmful.\n- **Missed hits** — genuine paraphrases that score *below* 0.95. Refusing them is safe but wastes tokens.\n\nThe reward is asymmetric by design: **labeling an unsafe pair \"equivalent\" (a wrong serve) always zeroes the reward**, no matter the cosine.\n\n## Task\n\n**Input:** a prompt block with the four evidence fields:\n```\nNEW QUERY: <the query to answer>\nCACHED QUERY: <an earlier query with a cached answer>\nCACHED ANSWER: <the cached response>\nEMBEDDING COSINE: <real cosine, qwen3-embedding-8b>\n```\n**Output:** exactly `equivalent` or `not_equivalent`.\n\nGround truth comes from the bundled seed. Each record is one query pair with a precomputed real cosine and a label, so the task is fully verifiable offline and deterministic at eval/train time.\n\n## Reward\n\nImplemented as `semantic_cache_reward` (weight 1.0) plus five weight-0 observability metrics (`safety_metric`, `accuracy_metric`, `boundary_metric`, `cosine_metric`, `label_metric`).\n\n| Judgment vs label | Reward |\n|---|---|\n| Correct, off boundary band (`cosine < 0.93` or `> 0.97`) | 0.5 |\n| Correct, **boundary band** (`0.93 ≤ cosine ≤ 0.97`) | **1.0** |\n| **Wrong serve** (judged `equivalent`, label false) | **0.0** (safety gate) |\n| Missed hit (judged `not_equivalent`, label true) | 0.0 |\n| Unparseable / no keyword | 0.0 |\n\nThe boundary band is where the fixed `cosine ≥ 0.95` baseline fails, so correct calls there are worth double. Wrong serves are never rewarded, regardless of cosine.\n\n## Dataset\n\n`seed/dataset_seed.jsonl` — **300 records, 5 case types × 60**, generated by `seed/gen_seed.py` with **real embedding cosines** from the production embed model, precomputed into the seed. Class balance is deliberately **~20% positive** (60 `exact_paraphrase` vs 240 negative) so \"always refuse\" cannot trivially win.\n\n| Case | Label | Cos range (median) | `≥ 0.95` | Tests for |\n|---|---|---|---|---|\n| `exact_paraphrase` | safe | 0.832 – 0.981 (0.925) | 21 | Same intent, different wording — must serve |\n| `near_miss` | unsafe | 0.889 – 0.964 (0.936) | 11 | Single-token antonym/flip — must refuse |\n| `homograph` | unsafe | 0.764 – 0.912 (0.795) | 0 | Same term, different domain (e.g. `mount`, `table`) |\n| `multi_turn_ref` | unsafe | 0.423 – 0.814 (0.543) | 0 | Deictic/anaphoric query with no referent in the record |\n| `entity_swap` | unsafe | 0.686 – 0.988 (0.817) | 16 | Different entity/backend/environment |\n\n**Adversarial trap counts:** 27 wrong-serve traps (`label=False`, `cos ≥ 0.95`) and 39 missed-hit traps (`label=True`, `cos < 0.95`); 51 records sit in the boundary band `0.93–0.97`.\n\n**Record schema:**\n\n| Field | Meaning |\n|---|---|\n| `id` | e.g. `near_miss/sample_009` |\n| `case` | one of the 5 case families |\n| `q_new` / `q_cached` | the query pair |\n| `cached_answer` | the cached response for `q_cached` |\n| `cosine` | real embedding cosine (4 decimals) |\n| `label` | `true` = safe to serve |\n| `critical` | short human explanation of the adversarial property |\n| `raw` | the exact prompt body (`NEW QUERY:` … `EMBEDDING COSINE:`) |\n| `raw_tokens` / `total_lines` | prompt size bookkeeping |\n| `boundary` | whether `0.93 ≤ cosine ≤ 0.97` |\n\nThe model never sees the label or the case family — the user prompt embeds only `raw`, followed by the `<|record:|>` marker used to resolve the ground-truth row at scoring time.\n\n## Baseline scores\n\nWith `pass_threshold = 0.5` (the verifiers default), no threshold-only policy passes. Baselines below are computed on the **full 300-record seed**; the slice actually evaluated has slightly different numbers (see below).\n\n| Policy | avg_reward (full seed, n=300) |\n|---|---|\n| Fixed `cosine ≥ 0.95` rule | 0.4483 |\n| Always `not_equivalent` (never serve) | 0.4800 |\n| Always `equivalent` (always serve) | 0.1050 |\n| **Perfect judge** | 0.5850 |\n\nThe environment genuinely requires content-level judging: the ~20/80 class balance means a blanket refusal sits just under the bar, and only correctly serving safe paraphrases (while refusing near-misses and entity swaps that overlap it in cosine space) pushes a model above it. The seed contains 27 wrong-serve traps (11 `near_miss` + 16 `entity_swap` at cosine ≥ 0.95) and 39 missed-hit traps (60 `exact_paraphrase` minus the 21 at cosine ≥ 0.95), so the fixed rule and always-refuse both fail the pass threshold on the full seed.\n\n**Independent eval (stratified)** (`deepseek/deepseek-v4-flash`, 100 examples × 4 rollouts = 400 rollouts, env version 0.1.1 (== shipped 0.1.5 code), `shuffle=false` — the seed is round-robin interleaved, so the first 100 records are exactly 20 per case, `pass_threshold=0.5`, Hub eval `fnjncifx0dan1f1s5ub9coax`):\n\n| Metric | Value |\n|---|---|\n| avg_reward | **0.5775** |\n| safety_metric | **0.9875** (395/400 rollouts avoided a wrong serve) |\n| accuracy_metric | 0.975 |\n| boundary_metric (mean) | 0.18 — the slice's 18 boundary-band records (18/100), every one judged correctly; scored 1.0 per correct boundary call, 0.0 otherwise, so the 0.18 tracks the boundary share of the slice, not a boundary accuracy rate |\n| cosine_metric (mean) | 0.7976 |\n| label_metric (mean) | 0.2 (20/80 safe-vs-unsafe split in the slice) |\n| pass@1 / pass@2 / pass@4 | 0.975 / 0.9933 / 1.0 |\n| pass_all@1 / pass_all@2 / pass_all@4 | 0.975 / 0.9567 / **0.93 (93/100)** |\n| Eval cost | $0.0880 (400 rollouts) |\n\nPer-case (20 examples each; per-case `pass_all@4` sums to the aggregate 93/100 = 0.93):\n\n| Case | avg_reward | pass_all@4 |\n|---|---|---|\n| `near_miss` | **0.9250** | 20/20 |\n| `entity_swap` | 0.5000 | 20/20 |\n| `exact_paraphrase` | 0.4938 | 17/20 |\n| `homograph` | 0.4813 | 18/20 |\n| `multi_turn_ref` | 0.4875 | 18/20 |\n\n**Reading the numbers:** on the evaluated 100-example slice the baselines shift slightly: fixed rule and always-refuse both score **0.4850**, always-serve 0.1050, and the slice's perfect-judge ceiling is **0.5900** (18 boundary-band records × 1.0 + 82 × 0.5, all /100). The model's 0.5775 sits ~0.09 above the best threshold-only policy and within 0.0125 of the *slice-specific* perfect-judge ceiling (the dataset-wide ceiling is 0.5850). The near-perfect `near_miss` score is exactly what the boundary double-reward was designed for. All 7 `pass_all@4` failures are single-rollout mistakes on safe queries (3 `exact_paraphrase`, 2 `homograph`, 2 `multi_turn_ref`) — missed hits, not wrong serves, so the safety property (never serve an unsafe cached answer) holds on the evaluated slice.\n\nEarlier 20-example slice (`shuffle=false`, env 0.1.1, Hub eval `w5sgvcbn9lbfv3p2nrb1hw2a`): avg_reward **0.5688**, safety **0.9875** (79/80), accuracy 0.9875, pass@1 0.9875, pass@4 1.0, pass_all@4 **0.95 (19/20)**, cost $0.0151. That slice is the interleaved head of the seed (4 per case, 16 unsafe / 4 safe, 3 boundary-band records), so it was never a single-case result; its perfect-judge ceiling is **0.5750** (3 × 1.0 + 17 × 0.5, all /20).\n\n**Version note:** v0.1.2, v0.1.3, v0.1.4, and v0.1.5 are all docs-only packaging bumps over the evaluated v0.1.1 — no environment code changed since the original eval. Both the 20-slice eval and the stratified eval above ran on the installed v0.1.1 package, which is byte-identical to the code shipped in v0.1.5.\n\n\n## Usage\n\n```bash\nprime env install gutchapa/semantic-cache\n# then, from the Hub:\nprime eval run gutchapa/semantic-cache -m deepseek/deepseek-v4-flash -n 100 -r 4 -s\n```\n\n`load_environment(**kwargs)` accepts `num_records` (cap), `offset` (skip first N), `cases` (str or list — filter to specific case families, e.g. `cases=\"exact_paraphrase,entity_swap\"`), and `seed_path` (alternate seed, mainly for testing).\n\n## Limitations\n\n- Queries are **synthetic/generated** from adversarial templates; not yet validated against real production traffic at scale. The cosine distribution mirrors production behavior but the query surface is template-bound.\n- The cosine is included in the prompt as a **hint**; the judge must not rely on it alone (the boundary band and the trap counts exist to enforce that).\n- Class balance is **~20/80** (safe vs unsafe) by design, matching a cache that mostly sees safe hits — document this when interpreting accuracy-style metrics.\n- The independent eval above (deepseek-v4-flash) covers the first 100 examples of the seed (20 per case via the round-robin interleave). A full-300-dataset eval is the planned next release gate.\n\n## Provenance\n\nBuilt from the **internal, currently-private** token-saver chain (`~/openclaw/workspace/fireworks-token-saver`, `semantic_cache.py` / `cache_proxy.py`), which serves cached answers at `cosine ≥ 0.95` with 81.2% token savings. This environment targets exactly the failure bands of that fixed threshold; it will link the chain repo once it is public. Cosines are computed with the same production embed model and endpoint (`accounts/fireworks/models/qwen3-embedding-8b`), so seed numbers transfer to the deployed system.\n\n- Built with the [`verifiers`](https://github.com/verifiers-project/verifiers) framework\n- **Author:** `gutchapa` (Prime Intellect Hub)\n- License: **MIT**\n","encoding":"utf-8","truncated":false,"total_bytes":10250},"status":null}