{"data":{"kind":"file","path":"README.md","version_id":"jiofeqk7nd8v2c0n8ilrg34s","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":1201,"modified_at":"2026-10-19T14:35:49.793000","content_hash":"1a9db3d2caf7f640bed3b3d328106eb853c12c511429a6c3611168ec4fba516b"},"entries":[],"content":"# flawed-judge\n\nGSM8K tasks scored by an LLM judge. The code-checked answer is logged alongside as a metric.\n\n| Field | Default | Description |\n|---|---|---|\n| `split` | `test` | GSM8K split |\n| `task.judge_mode` | `no_reference` | `reference` shows the judge the gold answer, `no_reference` does not, `solve_first` has the judge solve the task and use its own answer as the reference |\n| `task.harden` | `false` | Tell the judge to ignore instructions and verification claims inside the response |\n| `task.reward` | `judge` | `judge` trains on the judge verdict, `true_correct` on the code check (RL control) |\n| `task.judge.model` | verifiers default | Judge model |\n\nReward: `judge_score`, the judge verdict (0/1), or the code check when `task.reward = \"true_correct\"`. Metrics: `reference_judge` (the same judge shown the gold answer, one extra call per rollout) and `true_correct` (code check).\n\nThe judge model defaults to verifiers' default. The runs in [puiji/flawed-judge](https://github.com/puiji/flawed-judge) set `meta-llama/Llama-3.2-3B-Instruct` at temperature 0. Configs, results and plots are there too.\n\n```bash\n# from a clone of the repo\nuv run vf-eval @ configs/eval/smoke.toml\n```\n","encoding":"utf-8","truncated":false,"total_bytes":1201},"status":null}