{"data":{"kind":"file","path":"README.md","version_id":"um4g4p0o58fiozwdknb84kyc","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2523,"modified_at":"2026-09-07T14:12:55.889000","content_hash":"efb963fe9b6720f2f06becaf7969c0305fb799ec5dbb4c64b0612322cc28e7fc"},"entries":[],"content":"# Recommendation System Offline Evaluator v1.0.1\n\nEvaluates LLM ability to design, implement, and analyze offline evaluation metrics for recommendation systems.\n\n## Overview\n\nThis environment tests comprehensive knowledge of recommendation system evaluation across 10 diverse tasks covering:\n\n### Evaluation Dimensions\n\n1. **Ranking Metrics** — NDCG@K, MAP@K, MRR, Hit Rate@K with proper discounting and edge case handling\n2. **Beyond-Accuracy Metrics** — Intra-List Diversity, Catalog Coverage, Novelty, Serendipity\n3. **Statistical Testing** — Paired t-test, Wilcoxon signed-rank, Bootstrap CI, Cohen's d effect size\n4. **Temporal Evaluation** — Time-aware splits, user segmentation, popularity/position bias detection\n5. **Composite Quality Scoring** — Weighted multi-dimension evaluation with fairness dimension\n6. **Session-Based Evaluation** — Session NDCG, CTR prediction, repeat consumption penalty\n7. **Multi-Stakeholder Evaluation** — Consumer, provider, platform metrics with Pareto analysis\n8. **Bias Detection** — Gini coefficient, JS divergence, equal opportunity, Lorenz curves\n9. **Cross-Validation** — Temporal CV, user-stratified CV, nested CV for hyperparameter tuning\n10. **Evaluation Dashboard** — Executive summary, heat maps, error analysis, auto-recommendations\n\n### Scoring Rubric\n\n| Metric | Weight | Description |\n|--------|--------|-------------|\n| Code Quality | 20% | Structure, type hints, docstrings, imports, main block |\n| Completeness | 20% | Coverage of all requested components and concepts |\n| Implementation Correctness | 20% | Valid syntax, mathematical operations, proper logic flow |\n| Analysis Depth | 15% | Explanations, tradeoffs, statistical concepts, interpretations |\n| Edge Case Robustness | 15% | Zero/empty handling, division safety, try/except, validation |\n| Best Practices | 10% | Proper methodology, multi-dimension evaluation, appropriate libraries |\n\n### Skills Tested\n\n- Information retrieval evaluation (NDCG, MAP, MRR)\n- Beyond-accuracy evaluation (diversity, novelty, serendipity, coverage)\n- Statistical hypothesis testing and effect size estimation\n- Bias detection and fairness evaluation\n- Cross-validation methodology for recommendation data\n- Multi-stakeholder system evaluation\n- Evaluation dashboard and reporting\n\n## Usage\n\n```python\nimport verifiers as vf\n\nenv = vf.load_environment(\"olgaharri/recommendation-system-offline-evaluator\")\n```\n\n## Requirements\n\n- Python 3.10+\n- verifiers\n- datasets\n- numpy (for code generation tasks)\n","encoding":"utf-8","truncated":false,"total_bytes":2523},"status":null}