{"data":{"kind":"file","path":"README.md","version_id":"h5j18mm93uu9hac63aj6vzq7","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2633,"modified_at":"2026-05-05T01:02:35.597000","content_hash":"50e4c28f58730d2ab2a39e086537d3f76a93f02d84cfcd4fef261c43215a1dda"},"entries":[],"content":"# spdx-license-classify\n\nSingle-turn classification environment. Given the first ~600 characters of a\nsoftware license, the model must return the canonical\n[SPDX license identifier](https://spdx.org/licenses/) — for example `MIT`,\n`Apache-2.0`, `GPL-3.0-only`.\n\n### Overview\n- **Environment ID**: `spdx-license-classify`\n- **Short description**: Identify the SPDX license identifier from the opening of a license text.\n- **Tags**: single-turn, classification, license, spdx, compliance, eval\n\n### Datasets\n- **Primary dataset**: Live download of the SPDX standard license list, sourced\n  from [`spdx/license-list-data`](https://github.com/spdx/license-list-data)\n  (CC0).\n- **Source links**: `licenses.json` index + per-license `details/<id>.json`\n  bodies on `raw.githubusercontent.com`.\n- **Split sizes**: Eval-only. `max_licenses` arg (default `80`) caps the\n  number of texts pulled; the full canonical list has ~640 non-deprecated\n  identifiers.\n\n### Task\n- **Type**: single-turn\n- **Output format**: free text — the parser extracts the first SPDX-shaped\n  token (`[A-Za-z0-9][A-Za-z0-9.+-]*`) and strips common preludes such as\n  `SPDX identifier:`.\n- **Rubric overview**:\n  - `exact_id_match` (weight `1.0`) — exact equality with the gold SPDX ID,\n    case-insensitive.\n  - `family_match` (weight `0.0`, informational) — `1.0` when the version\n    family prefix matches (e.g. `GPL-3.0-only` vs `GPL-3.0-or-later`),\n    useful for diagnosing near-misses.\n\n### Quickstart\n\n```bash\nuv run vf-install spdx-license-classify\nuv run vf-eval -s spdx-license-classify -m gpt-5-mini -n 5 -r 3\n```\n\n### Environment Arguments\n\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `max_licenses` | int | `80` | Cap on dataset size pulled from SPDX. Lower for cheap smoke tests; raise toward `640` for richer eval. |\n\n### Metrics\n\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` | Equals `exact_id_match` (the only weighted reward). |\n| `exact_id_match` | 1.0 iff prediction equals gold SPDX ID (case-insensitive). |\n| `family_match` | 1.0 when the family prefix matches (e.g. both `GPL-*`). Diagnostic only. |\n\n### Notes\n\n- Snippets are deliberately truncated to **600 chars** so the task is\n  non-trivial: the model must rely on early-text fingerprints (preamble\n  wording, copyright clause, etc.) rather than scanning for a\n  `SPDX-License-Identifier:` header further down.\n- No credentials required — the SPDX corpus is public CC0 data.\n- The corpus is downloaded on each `load_environment` call via `urllib`;\n  no local caching beyond the in-memory `datasets.Dataset` returned to\n  verifiers.\n","encoding":"utf-8","truncated":false,"total_bytes":2633},"status":null}