{"data":{"kind":"file","path":"README.md","version_id":"nie8s32bdhbcysyhfu9xjkxo","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":1892,"modified_at":"2025-09-07T19:03:35.272000","content_hash":"803a1e12c0bb4afa0a7d55984971e25d6845e467b2d6e1cf7fabcf81aa2ba3cb"},"entries":[],"content":"# cuda_llm\n\n### Overview\n- **Environment ID**: `cuda_llm`\n- **Short description**: RL environment for evaluating models on CUDA kernel generation tasks\n- **Tags**: cuda, kernel-generation, compilation, bytedance, prime-intellect\n\n### Datasets\n- **Primary dataset(s)**: ByteDance CUDA LLM dataset with real CUDA kernel generation problems\n- **Source links**: [ByteDance-Seed/cudaLLM-data](https://huggingface.co/datasets/ByteDance-Seed/cudaLLM-data/tree/main)\n- **Split sizes**: Variable based on parquet file contents\n\n### Task\n- **Type**: single-turn\n- **Parser**: CudaCodeParser (custom parser for extracting CUDA code from model responses)\n- **Rubric overview**: \n  - Compilation reward (1.0x) - code compiles and runs successfully\n  - Code validation reward (0.3x) - has CUDA keywords, proper structure, valid syntax\n  - Format reward (0.1x) - code properly extracted from markdown blocks\n\n### Quickstart\nRun an evaluation with default settings:\n\n```bash\nuv run vf-eval cuda_llm\n```\n\nConfigure model and sampling:\n\n```bash\nuv run vf-eval cuda_llm -m gpt-4o -n 20 -r 3 -t 1024 -T 0.7\n```\n\n### Environment Arguments\n\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `max_examples` | int | `-1` | Limit on dataset size (use -1 for all) |\n\n### Metrics\n\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` | Main scalar reward (weighted sum of compilation, validation, and format) |\n| `compilation_reward` | Binary reward for successful CUDA code compilation |\n| `code_validation_reward` | Reward for code structure and syntax validity |\n| `format_reward` | Reward for proper code extraction and formatting |\n\n### Testing\n\ntest the environment setup:\n```bash\npython test_environment.py\n```\n\ntest rewards on specific outputs:\n```bash\npython test_rewards.py --file some_output.txt\npython test_rewards.py --interactive\npython test_rewards.py 'your code here'\n```\n\n","encoding":"utf-8","truncated":false,"total_bytes":1892},"status":null}