{"data":{"kind":"file","path":"README.md","version_id":"tdyqdg7t8q350kacb2hbfk6r","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2654,"modified_at":"2026-06-22T08:54:10.915000","content_hash":"6d2f8d16772ea552892af000779f969b0a302682ca5f3bd6569ac753dede8f56"},"entries":[],"content":"# linkd-search-llm\n\nVerifiers environment for the **linkd-search** task on **REAL user queries** (no gold filters):\ntranslate a natural-language people-search query into a **MongoDB `find` filter** (raw JSON) for\nthe **live** `Berkeley.profilematch` collection (269k profiles), scored by an **LLM judge** over\nthe live top-K retrieval.\n\nThis is the *no-gold* sibling of `freesolo-co/linkd-search` (which trains on oracle filters with\ndeterministic retrieval-F1). Here the queries are real, there are no oracle filters/gold_ids, so\nthe learning signal comes from the LLM judge + live Mongo execution.\n\n- **Input**: a real NL query (e.g. \"head of sales/bd leaders in the US with buying power\").\n- **Output**: a raw JSON Mongo filter using only the allowed schema fields/operators.\n- **Reward** (weighted): `0.55*llm_judge + 0.15*contract + 0.15*safety + 0.15*selectivity`\n  - `llm_judge` (0-1): a strong instruct judge (Fireworks `gpt-oss-120b`, OpenAI-compatible)\n    scores how well the **top-K profiles returned from live Mongo** satisfy the query.\n  - `contract` (1/0): the filter parses to JSON, passes the allowlist validator, AND executes on\n    live Mongo without error.\n  - `safety` (1/0): the filter uses ONLY allowed fields/operators with valid structure.\n  - `selectivity` (0-1): the result set is neither empty nor over-broad.\n  A **deterministic pre-gate** (parse + safety + execute) returns `0.0` WITHOUT a judge call on\n  garbage. The judge runs **once per rollout** (a single combined call over the top-K profiles);\n  Mongo + judge results are cached per `(query, filter)`. On a persistent judge ERROR the\n  `llm_judge` term returns a neutral `0.3` for a valid non-empty retrieval (it can't fabricate a\n  real score) so an outage doesn't zero the batch.\n\n## Live-resource config (passed via the run's `worker_env`; never hardcoded)\n\n- `LINKD_MONGO_URL` — mongodb URI for `Berkeley.profilematch` (**required**).\n- `LINKD_JUDGE_AUTH` — Fireworks API key for the judge. This worker_env-safe name is used by runs\n  (the flash schema rejects `*_API_KEY` names in `worker_env`); `LINKD_JUDGE_API_KEY` /\n  `FIREWORKS_API_KEY` are also accepted for local/standalone use.\n- `LINKD_JUDGE_MODEL` — optional judge model override (default `accounts/fireworks/models/gpt-oss-120b`).\n- `LINKD_JUDGE_BASE_URL` — optional OpenAI-compatible base url (default Fireworks).\n\n`load_environment(max_train=-1, max_eval=-1)` returns a `vf.SingleTurnEnv`. Thinking vs\nnon-thinking is handled by the trainer (chat-template `enable_thinking` + `<think>` stripping);\nthe reward is identical either way (`extract_filter` strips a leading `<think>…</think>`).\n","encoding":"utf-8","truncated":false,"total_bytes":2654},"status":null}