{"data":{"kind":"file","path":"README.md","version_id":"utqlfdb9zvir0e707mxzu4de","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":13647,"modified_at":"2026-09-02T17:39:40.422000","content_hash":"cefeb976e78c73e39ae2a8bfd2d5d89c2f1faf0bd0b080677b07f9538f241718"},"entries":[],"content":"# faithful-compression\n\nA `verifiers`-spec **single-turn RL environment** for **fidelity-gated log compression**: given a raw, noisy log excerpt, the model must produce a compressed log that keeps every critical line (errors, failures, exceptions) while minimizing size.\n\nEnvironment: **`gutchapa/faithful-compression`** · version **0.1.7** · license **MIT** · published as a **public** Hub environment.\n\n---\n\n## Why this task\n\nAgentic coding tools re-send raw tool output (build logs, test-runner chatter, repeated errors) on every turn. A model that compresses noisy logs *faithfully* — dropping repetition and noise but never a critical line — directly cuts inference cost in long-running agent loops.\n\nThis environment is the RL half of that problem: it scores a model's compression against a ground-truth set of critical lines, with a hard fidelity gate so compression can never be bought at the expense of information.\n\n---\n\n## Quick facts\n\n| Field | Value |\n|---|---|\n| Framework | `verifiers` `SingleTurnEnv` (Prime Intellect) |\n| Env ID | `gutchapa/faithful-compression` |\n| Version | 0.1.7 |\n| License | MIT |\n| Dataset | `seed/dataset_seed.jsonl` — 200 records, 5 case types × 40 |\n| Reward | `fidelity_and_compression_reward` (two-gate, fidelity dominates) |\n| Install | `prime env install gutchapa/faithful-compression` |\n\n---\n\n## Task definition\n\n**Input:** a raw server-log excerpt (build output, test-runner chatter, repeated errors).\n\n**Output:** a compressed log such that:\n\n1. **Every critical line survives** — the specific error/failure/exception lines needed for debugging.\n2. **Redundancy is removed** — runs of identical messages collapse; noise is filtered.\n\nGround truth comes from the bundled seed. Each record is one raw log with its critical lines marked, so the task is fully verifiable offline.\n\n---\n\n## Reward: two gates, fidelity dominates\n\nThe reward is deliberately asymmetric: **losing a critical line always zeroes the reward, no matter how good the compression**. Implemented as `fidelity_and_compression_reward` (weight 1.0 in the verifiers rubric) plus three weight-0 observability metrics (`fidelity_metric`, `compression_ratio_metric`, `reference_ratio_metric`).\n\n### Gate 1 — Fidelity (hard, binary)\n\n- `distinct_critical_contents(record)` derives the required set: for each ground-truth critical line, strip the case's adversarial **`prefix_regex`** (e.g. `timestamp`: `^\\s*\\d{1,2}:\\d{2}:\\d{2}\\.?\\d*\\s*`; `req_id`: `^\\s*req-[\\w-]+\\s*:?\\s*`), then whitespace-normalize (`\\s+` → single space).\n- The candidate output is split into lines and given the same strip + normalize treatment.\n- **The gate passes iff every required content is present in the candidate line set.** Any miss → `reward = 0.0`, regardless of compression.\n\n### Gate 2 — Compression score (only scored after the gate passes)\n\n- **Token estimator** — `estimate_tokens(text) ≈ max(1, len(text) // 4)` (≈4 chars/token), matching the seed's token accounting so scoring is deterministic inside the eval container.\n- **Model ratio** — `ratio = estimate_tokens(candidate) / estimate_tokens(raw)` (this is `compression_ratio_metric`).\n- **Reference ratio** — each record carries a `reference_ratio`: the ratio a deterministic baseline compressor achieves on that same record (`reference_ratio_metric`).\n- **Ideal ratio** — each record carries `ideal_ratio`: the ratio of a perfect compressor that emits exactly the distinct critical lines.\n- **Score:**\n\n```\nheadroom = reference_ratio − ideal_ratio\nspan     = headroom  if headroom > max(1e-4, ideal_ratio · 1e-3)   # real headroom exists\n         = reference_ratio                                        # reference already at the floor\nscore    = clamp((reference_ratio − ratio) / span, −1.0, 1.0)\n```\n\n- Tying the reference scores exactly **0**; only strictly-more-compact output scores positive (e.g. collapsing identical-message runs). Worse than the reference scores negative, clamped to −1.\n- If the reference is already within rounding noise of the ideal floor, the span collapses to the reference itself so the environment never rewards impossible headroom.\n\n### Final reward\n\n```\nreward = compression_score  if fidelity_gate passes\n       = 0.0               otherwise\n```\n\nFidelity is a hard gate, not a soft term: 0.7236 average reward with a 1.0 fidelity rate means the model earned that score *while never dropping a critical line*.\n\n---\n\n## Dataset\n\n`seed/dataset_seed.jsonl` — 200 records, 5 case types × 40, generated from an adversarial stress-test suite (see Origin).\n\n| Case | Tests for |\n|---|---|\n| `timestamp` | Identical messages with varying timestamp prefixes — should collapse as duplicates |\n| `req_id` | Identical messages with varying request-ID prefixes — should collapse |\n| `line_num` | Identical messages with sequential line-number prefixes — should collapse |\n| `false_positive` | Distinct error messages sharing a first word (e.g. multiple `\"Error: ...\"` lines) — should **NOT** be merged |\n| `mixed_noise` | Signal interleaved with high-cardinality noise — tests filter-before-group ordering |\n\n**Record schema:**\n\n| Field | Meaning |\n|---|---|\n| `id` | e.g. `timestamp/sample_000` |\n| `case` | one of the five above |\n| `raw` | input log text |\n| `critical` | list of `{line, text}` ground-truth lines that must survive |\n| `distinct_critical` | deduplicated normalized critical contents (what the fidelity gate checks) |\n| `prefix_regex` | adversarial per-case prefix to strip before comparison |\n| `ideal_ratio` | perfect-compressor ratio (4 decimals) |\n| `reference_ratio` | baseline-compressor ratio (full precision) |\n| `raw_tokens` | `estimate_tokens(raw)` |\n| `total_lines` | line count of `raw` |\n\nThe model never sees the critical fields — the user prompt embeds only the raw log between `--- LOG START ---` / `--- LOG END ---` markers, followed by `Compressed log:`.\n\n---\n\n## Evaluation\n\n**Independent eval (stratified)** — `deepseek/deepseek-v4-flash`, 50 examples × 4 rollouts (200 rollouts), env version 0.1.3 (== shipped 0.1.7 code), `shuffle=true`, `shuffle_seed=42`, `pass_threshold=0.5`. Hub eval `wxly8xeyz3ydk13hei8xufn0`:\n\n| Metric | Value |\n|---|---|\n| avg_reward | **0.4558** |\n| fidelity_metric | **0.985** (197/200 rollouts passed the gate) |\n| compression_ratio_metric (mean) | 0.1176 |\n| reference_ratio_metric (mean) | 0.2143 |\n| pass@1 / pass@2 / pass@4 | 0.695 / 0.8533 / 0.94 |\n| pass_all@1 / pass_all@2 / pass_all@4 | 0.695 / 0.5367 / **0.36 (18/50)** |\n| Eval cost | $0.2502 (200 rollouts) |\n\nPer-case (`--shuffle --shuffle-seed 42` sample of 50 from the 200-record seed; all five case families covered; per-case `pass_all@4` sums to the aggregate 18/50 = 0.36):\n\n| Case | n | avg_reward | pass_all@4 |\n|---|---|---|---|\n| `timestamp` | 12 | 0.6050 | 6/12 |\n| `req_id` | 9 | 0.5848 | 4/9 |\n| `line_num` | 7 | 0.9959 | 7/7 |\n| `false_positive` | 11 | **-0.0956** | 1/11 |\n| `mixed_noise` | 11 | 0.3952 | 0/11 |\n\n**Reading the numbers:** the model compressed inputs to ~11.8% of their original token count on average (`compression_ratio_metric` 0.1176), against a baseline compressor at ~21.4% (`reference_ratio_metric` 0.2143) — a ~1.8× size reduction over the baseline. Three rollouts over-compressed and dropped a critical line (fidelity 0 → reward 0); several rollouts under-compressed so far above the baseline ratio that the reward went negative (min -1.0). Per case the picture separates cleanly: `line_num` is essentially perfect (0.9959, 7/7) and `timestamp` / `req_id` are solid, while `false_positive` (**−0.0956**) and `mixed_noise` (0.3952) carry the difficulty — and the `false_positive` result is the sharpest finding in the eval, not a fluke.\n\n**Why the `false_positive` result matters:** the pipeline's Stage 4 grouping heuristic (see Origin) fails by merging distinct `Error:` lines that share a first word — and this stratified eval shows DeepSeek, a full LLM with no rule-based grouping step at all, reproducing that exact failure on the same case family. \"Distinct messages sharing surface-level structure\" is evidently a genuinely hard class of compression error, one that survives the jump from a symbolic heuristic to a neural compressor rather than being an artifact of the original pipeline's simplicity. That makes `false_positive` a task worth training on, not a weakness to explain away.\n\nEarlier 20-example slice (`shuffle=false`, env 0.1.3, Hub eval `l0z6py1eoyrqlomga66f9lks`): avg_reward **0.4978**, fidelity **0.975** (78/80), pass@1 0.7875, pass@4 1.0, pass_all@4 **0.45 (9/20)**, cost $0.0734. That slice covered only the `timestamp` case (seed is ordered timestamp → req_id → line_num → false_positive → mixed_noise, 40 each; first 20 = timestamp).\n\nReference eval (pre-rename `rtk-compression`, `deepseek/deepseek-chat`, run hash `159534dc`, Hub eval `ypvsne4gjrnfwpxz8yz434tm`): avg_reward **0.7236**, fidelity **1.0**, pass@1 0.9625, pass@4 1.0, pass_all@4 0.85 (17/20), cost $0.0347. v4-flash compresses less aggressively, which is why its reward is lower.\n\n**On the headline drop:** the earlier timestamp-only teasers (0.45 for v4-flash, 0.85 for deepseek-chat pass_all@4) were measuring the easy slice. The stratified pass_all@4 of **0.36** across all five case families is not the environment regressing — it is the honest full-spread number, and it is the point: the task separates cases a model handles well (`line_num` 7/7) from cases it does not (`false_positive` 1/11, `mixed_noise` 0/11). An environment every model aces on every case has no training signal left in it; reproducible difficulty variance across case families is what makes this useful as an RL training target.\n\n**Version note:** v0.1.3, v0.1.4, v0.1.5, v0.1.6, and v0.1.7 are all docs-only packaging bumps over the evaluated v0.1.2 — no environment code changed since the original eval. Both the 20-slice eval and the stratified eval above ran on the installed v0.1.3 package, which is byte-identical to the code shipped in v0.1.7.\n\n\n---\n\n## Coverage caveat (read this)\n\n## Coverage\n\nThe shuffled-50 stratified eval above covers **all five case families**. Per-case example counts are uneven (12/9/7/11/11) because they come from a reproducible shuffle (`--shuffle --shuffle-seed 42`), not a uniform 10-per-case sample.\n\n| Case | In official eval | Other coverage |\n|---|---|---|\n| `timestamp` | ✅ 12 examples / 48 rollouts | — |\n| `req_id` | ✅ 9 examples / 36 rollouts | — |\n| `line_num` | ✅ 7 examples / 28 rollouts | — |\n| `false_positive` | ✅ 11 examples / 44 rollouts | pipeline-level adversarial stress-test of the Stage-4 grouping heuristic (see Origin) |\n| `mixed_noise` | ✅ 11 examples / 44 rollouts | — |\n\n**Status:** all five case families now have environment-eval numbers. The `false_positive` case is where the model reproduces the pipeline's own Stage-4 failure mode — merging distinct `Error:` lines that share a first word (a documented finding, see the eval reading above); `mixed_noise` sits at the compression gate. A uniform 10-per-case stratified sample and a full-200-dataset eval are the planned next release gates.\n\n\n---\n\n## Validation history\n\nAn earlier eval run scored **0.5346** (run `e36a84a2`) due to a bug in `_extract_candidate_text`: for non-dict message objects it fell back to a `repr`/`str` representation instead of extracting the actual model output — silently scoring the wrong string. It was caught by treating a suspicious `fidelity_metric == 1.0` flatline as something to audit rather than trust, not by an external review. Fixed in v0.1.2; the fix was verified to reproduce all 80 honest scores. Buggy artifacts are preserved under `run1_pre_fix/` for provenance.\n\n---\n\n## Origin\n\nBuilt on top of an **internal, currently-private** 5-stage deterministic log-compression pipeline (`strip → filter → dedup → group → truncate`). This environment specifically targets the edge cases where that pipeline's **Stage 4 (grouping)** heuristic fails — the dataset was generated from an adversarial stress-test suite designed to break the first-word-prefix grouping key. The pipeline repo is not yet public; this section will link it once it is.\n\n---\n\n## Usage\n\n```bash\n# install (private env — requires your Hub auth)\nprime env install gutchapa/faithful-compression\n\n# run the reference eval\nprime eval run gutchapa/faithful-compression -m deepseek/deepseek-v4-flash -n 50 -r 4 --shuffle --shuffle-seed 42 -s\n```\n\n`load_environment(**kwargs)` accepts `cases` (str or list — filter to specific case families, e.g. `cases=\"timestamp,false_positive\"`) and `seed_path` (alternate seed, mainly for testing).\n\n---\n\n## Limitations\n\n- Dataset currently covers **synthetic/generated** log patterns; not yet validated against real production logs at scale.\n- Official eval coverage now spans **all five case families** via the shuffled-50 stratified eval (see Coverage caveat); a uniform stratified sample and a full-200-dataset eval are the planned next release gates.\n- Token accounting uses a deterministic `len // 4` estimator, not a real tokenizer — fine for relative scoring, not absolute token counts.\n- Fidelity is checked via normalized **content containment**, not semantic equivalence — a critical line paraphrased beyond its normalized form would be flagged as missing.\n- Env is **public** on the Hub; install with `prime env install gutchapa/faithful-compression`.\n\n---\n\n## Credits & license\n\n- **Author:** `gutchapa` (Prime Intellect Hub)\n- Built with the [`verifiers`](https://github.com/verifiers-project/verifiers) framework\n- License: **MIT**\n","encoding":"utf-8","truncated":false,"total_bytes":13647},"status":null}