{"data":{"kind":"file","path":"README.md","version_id":"ee2uk33oofbt9mfoihgaacnr","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":7390,"modified_at":"2025-10-23T05:01:05.937000","content_hash":"acfd94369afbdb7bec6628cef936f4b722a557b022c4eef6de8c90af082a9332"},"entries":[],"content":"# Code Vulnerability Remediation\n\nA tool-using RL environment for training and evaluating models on vulnerability detection and patching. Models analyze vulnerable code snippets, generate security patches, and validate fixes through automated testing.\n\n## Overview\n\nThis environment implements patch-and-test vulnerability remediation with executable validation, combining static analysis with runtime testing to ensure secure fixes maintain functionality.\n\n**Environment Type**: `ToolEnv` - Multi-turn environment with tool access\n**Task**: Identify vulnerabilities and generate validated patches for Python code\n**Tools**: Static security scanner, patch application and test executor\n**Reward Structure**: Test success + patch quality + security validation\n\n## Installation\n\nInstall the environment using the Prime CLI:\n\n```bash\nprime env install intertwine/sv-env-code-vulnerability\n```\n\nOr using pip directly:\n\n```bash\npip install sv-env-code-vulnerability\n```\n\n## Setup\n\n### API Keys Configuration\n\nSet your API keys as environment variables:\n\n```bash\n# OpenAI API Key (required for OpenAI models)\nexport OPENAI_API_KEY=\"your-openai-api-key\"\n\n# For persistent configuration\necho 'export OPENAI_API_KEY=\"your-key\"' >> ~/.bashrc\nsource ~/.bashrc\n```\n\n## Usage\n\n### With Verifiers Library\n\n```python\nimport verifiers as vf\n\n# Load the environment with tools enabled\nenv = vf.load_environment(\"intertwine/sv-env-code-vulnerability\", include_tools=True)\n\n# Evaluate a model\nresults = env.evaluate(\n    client=vf.OpenAIClient(),\n    model=\"gpt-5-mini\",\n    num_examples=10\n)\n\nprint(f\"Average reward: {results.stats['mean_reward']:.2%}\")\nprint(f\"Test pass rate: {results.stats.get('tests_passed_rate', 0):.2%}\")\n```\n\n### Quick Evaluation\n\nUse the verifiers CLI:\n\n```bash\n# Basic evaluation with tools\nvf-eval intertwine/sv-env-code-vulnerability \\\n  --model gpt-5-mini \\\n  --num-examples 10\n\n# Without tools (direct patching)\nvf-eval intertwine/sv-env-code-vulnerability \\\n  --model gpt-5-mini \\\n  --num-examples 10 \\\n  --include-tools false\n```\n\n### Training with Prime RL\n\n```toml\n[environment]\nid = \"intertwine/sv-env-code-vulnerability\"\nkwargs = {include_tools = true}\n```\n\n## Task Details\n\n### Input Format\n\nVulnerable Python code snippet:\n\n```python\ndef get_user_data(user_id):\n    query = f\"SELECT * FROM users WHERE id = {user_id}\"  # SQL injection vulnerability\n    return execute_query(query)\n```\n\n### Expected Output\n\nJSON object with patch and test results:\n\n```json\n{\n  \"diff\": \"--- a/code.py\\n+++ b/code.py\\n@@ -1,3 +1,3 @@\\n def get_user_data(user_id):\\n-    query = f\\\"SELECT * FROM users WHERE id = {user_id}\\\"\\n+    query = \\\"SELECT * FROM users WHERE id = ?\\\"\\n+    return execute_query(query, (user_id,))\",\n  \"tests_passed\": true,\n  \"explanation\": \"Fixed SQL injection by using parameterized query\",\n  \"patched_code\": \"def get_user_data(user_id):\\n    query = \\\"SELECT * FROM users WHERE id = ?\\\"\\n    return execute_query(query, (user_id,))\"\n}\n```\n\n### Available Tools\n\nWhen `include_tools=True`, the model has access to:\n\n1. **run_python_static_scan**: Heuristic SAST for risky constructs\n   - SQL concatenation detection\n   - Unsafe YAML loading\n   - Insecure randomness\n   - Command injection risks\n\n2. **run_patch_and_tests**: Apply patches and run validation\n   - Applies unified diff or full patched code\n   - Executes behavior tests\n   - Runs security regression tests\n   - Returns pass/fail status\n\n### Scoring\n\nThe reward function weights multiple components:\n\n- **Test Execution** (60%): Regression suite must pass\n- **Patch Similarity** (20%): Alignment with reference fix\n- **Test Consistency** (10%): Claimed vs actual test results\n- **Explanation Quality** (10%): Coverage of security concepts\n\n## Weights & Biases Logging\n\nThis environment supports automatic Weave tracing:\n\n```python\nimport weave\nimport verifiers as vf\n\n# Initialize Weave\nweave.init(project=\"vulnerability-repair\")\n\n# Load and evaluate\nenv = vf.load_environment(\"intertwine/sv-env-code-vulnerability\", include_tools=True)\nresults = env.evaluate(\n    client=vf.OpenAIClient(),\n    model=\"gpt-5-mini\",\n    num_examples=50\n)\n\n# Results automatically traced to W&B\n```\n\nConfigure via environment variables:\n- `WEAVE_PROJECT`: Set project name\n- `WEAVE_DISABLED`: Set to 'true' to disable logging\n- `WANDB_API_KEY`: Your W&B API key\n\n## Evaluation Approach\n\n### Metrics Tracked\n- **Vulnerability Detection Rate**: Identifying security issues\n- **Patch Success Rate**: Fixes that pass all tests\n- **Security Validation**: Confirmation vulnerability is resolved\n- **Code Quality**: Maintaining functionality while fixing issues\n- **Explanation Accuracy**: Understanding of vulnerability and fix\n\n### Example Evaluation Script\n\n```python\nimport verifiers as vf\nimport weave\n\nweave.init(project=\"vuln-repair-eval\")\n\nenv = vf.load_environment(\"intertwine/sv-env-code-vulnerability\", include_tools=True)\n\n# Evaluate across different vulnerability types\nresults = env.evaluate(\n    client=vf.OpenAIClient(),\n    model=\"gpt-5-mini\",\n    num_examples=100,\n    seed=42\n)\n\nprint(f\"Mean Reward: {results.stats['mean_reward']:.2%}\")\nprint(f\"Detection Rate: {results.stats.get('detection_rate', 0):.2%}\")\nprint(f\"Patch Success: {results.stats.get('patch_success', 0):.2%}\")\nprint(f\"Test Pass Rate: {results.stats.get('tests_passed_rate', 0):.2%}\")\n```\n\n## Performance Benchmarks\n\n| Model       | Detection | Patch Success | Tests Passed | Overall |\n|-------------|-----------|---------------|--------------|---------|\n| GPT-4o-mini | 85%       | 62%           | 71%          | 68%     |\n| GPT-4o      | 92%       | 78%           | 84%          | 82%     |\n\n## Vulnerability Types\n\nThe environment includes diverse vulnerability patterns:\n\n- **Injection Flaws**: SQL, command, LDAP injection\n- **Insecure Deserialization**: Pickle, YAML unsafe loading\n- **Cryptographic Issues**: Weak randomness, hardcoded keys\n- **Path Traversal**: Directory traversal vulnerabilities\n- **XXE/XML Issues**: External entity vulnerabilities\n- **Insecure Defaults**: Unsafe configurations\n\n## Dataset\n\n- **Vulnerable Snippets**: Real-world inspired Python vulnerabilities\n- **Reference Patches**: Security-validated fixes\n- **Test Suites**: Behavior and security regression tests\n- **Explanations**: Security rationale for each fix\n\n## Future Improvements\n\n- **Language Expansion**: Support for JavaScript, Java, Go vulnerabilities\n- **Complex Vulnerabilities**: Multi-file, cross-function security issues\n- **Fuzzing Integration**: Property-based testing for patch validation\n- **Performance Metrics**: Track fix impact on code performance\n- **Security Frameworks**: Map to OWASP Top 10, CWE classifications\n- **Incremental Repair**: Iterative refinement based on test feedback\n\n## Requirements\n\n- Python 3.12+\n- `verifiers>=0.1.4`\n- API key for model inference\n\n## About\n\nThis environment is part of the Open Security Verifiers suite - a collection of security and alignment RL environments using Prime Intellect's Verifiers framework. Each environment provides executable, programmatic rewards for training robust security-aware AI systems.\n\n## Support\n\nFor issues or questions:\n- Report issues on the [Prime Intellect Environments Hub](https://app.primeintellect.ai/dashboard/environments)\n- Check the [Security Verifiers GitHub repository](https://github.com/intertwine/security-verifiers)\n- Contact the Intertwine team\n","encoding":"utf-8","truncated":false,"total_bytes":7390},"status":null}