{"data":{"kind":"file","path":"README.md","version_id":"lwwdeywwykf3bohcpr7osh21","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":11026,"modified_at":"2026-02-28T02:07:06.022000","content_hash":"a6c5a6304f33ff7955689415341c29decb038cfab280aacc377a83aca5d6fcfa"},"entries":[],"content":"# Network Log Anomaly Detection (E1)\n\nA security-focused RL environment for training and evaluating models on network intrusion detection. Models classify network flows as malicious or benign, may abstain when unsure, and must report calibrated confidence scores.\n\n## Overview\n\nThis environment implements calibrated classification with abstention support and asymmetric costs, enabling realistic evaluation of network intrusion detection agents.\n\n**Environment Type**: `SingleTurnEnv` - One prompt, one response per example\n**Task**: Ternary classification of network logs (Malicious / Benign / Abstain)\n**Reward Structure**: Accuracy, JSON format compliance, calibration, and cost-sensitive penalties\n**Dataset**: IoT-23 network traffic with labeled malicious/benign connections\n\n## Dataset Access\n\n**Public Metadata**: Browse sampling information and dataset composition at:\n\n- <https://huggingface.co/datasets/intertwine-ai/security-verifiers-e1-metadata>\n\n**Full Dataset**: Private to prevent training contamination. Request access via:\n\n- [GitHub Issues](https://github.com/intertwine/security-verifiers/issues) with title \"Dataset Access Request: E1\"\n- Include: name, affiliation, research purpose, HuggingFace username\n\nThe public metadata repo includes detailed model cards explaining the privacy rationale and dataset composition.\n\n### Dataset Loading Strategies\n\nThis environment supports **multi-tiered dataset loading** for flexibility across different deployment scenarios:\n\n1. **Local datasets** (built with `make data-e1`)\n2. **HuggingFace Hub** (with `HF_TOKEN` authentication)\n3. **Synthetic fixtures** (for testing without data dependencies)\n\n#### Loading Modes\n\n```python\nimport verifiers as vf\n\n# Auto mode (default): Try local → hub → synthetic\nenv = vf.load_environment(\"sv-env-network-logs\")\n\n# Local only: Require local dataset\nenv = vf.load_environment(\"sv-env-network-logs\", dataset_source=\"local\")\n\n# Hub only: Load from HuggingFace\nenv = vf.load_environment(\"sv-env-network-logs\", dataset_source=\"hub\")\n\n# Synthetic only: Use test fixtures (no data needed)\nenv = vf.load_environment(\"sv-env-network-logs\", dataset_source=\"synthetic\")\n```\n\n#### Using Your Own HuggingFace Repository\n\nIf you've built and pushed datasets to your own HuggingFace repository:\n\n```python\nimport os\n\n# Configure custom repository\nos.environ[\"HF_TOKEN\"] = \"hf_your_token_here\"\nos.environ[\"E1_HF_REPO\"] = \"your-org/security-verifiers-e1-private\"\n\n# Load from your repository\nenv = vf.load_environment(\n    \"sv-env-network-logs\",\n    dataset_source=\"hub\",\n    max_examples=100\n)\n```\n\n**See [docs/user-dataset-guide.md](../../docs/user-dataset-guide.md) for instructions on building and pushing datasets to your own HuggingFace repository.**\n\n## Installation\n\nInstall the environment using the Prime CLI:\n\n```bash\nprime env install intertwine/sv-env-network-logs\n```\n\nOr using pip directly:\n\n```bash\npip install sv-env-network-logs\n```\n\n## Setup\n\n### API Keys Configuration\n\nBefore using this environment, you need to configure API keys for model inference and dataset access:\n\n1. **Set your API keys as environment variables**:\n\n   ```bash\n   # OpenAI API Key (required for OpenAI models)\n   export OPENAI_API_KEY=\"your-openai-api-key\"\n\n   # HuggingFace Token (optional, for IoT-23 dataset access)\n   export HF_TOKEN=\"your-huggingface-token\"\n   ```\n\n   Get your HuggingFace token from [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens)\n\n**Note**: Without the HF_TOKEN, the environment will fall back to using a synthetic dataset with limited examples.\n\n1. **For persistent configuration, add to your shell profile**:\n\n   ```bash\n   echo 'export OPENAI_API_KEY=\"your-key\"' >> ~/.bashrc\n   echo 'export HF_TOKEN=\"your-token\"' >> ~/.bashrc\n   source ~/.bashrc\n   ```\n\n## Usage\n\n### With Prime CLI (Recommended)\n\nThe easiest way to evaluate models on this environment is using the Prime CLI:\n\n```bash\n# Install the environment\nprime env install intertwine/sv-env-network-logs\n\n# Run evaluation with default dataset (1000 examples from HuggingFace)\nprime env eval sv-env-network-logs \\\n  -a '{\"dataset_name\":\"intertwine-ai/security-verifiers-e1\",\"max_examples\":100}'\n\n# Run with specific dataset split\nprime env eval sv-env-network-logs \\\n  -a '{\"dataset_name\":\"intertwine-ai/security-verifiers-e1\",\"max_examples\":100}' \\\n  --num-examples 10\n\n# Use synthetic dataset for quick testing (no HF_TOKEN required)\nprime env eval sv-env-network-logs \\\n  -a '{\"dataset_source\":\"synthetic\"}' \\\n  --num-examples 5\n```\n\n**Note**: By default, Prime uses meta-llama/llama-3.1-70b-instruct. Specify a different model with `--model`:\n\n```bash\nprime env eval sv-env-network-logs \\\n  -a '{\"dataset_name\":\"intertwine-ai/security-verifiers-e1\"}' \\\n  --model gpt-4o-mini \\\n  --num-examples 10\n```\n\n### With Verifiers Library\n\n```python\nimport os\nimport verifiers as vf\n\n# Load environment variables from .env file (if running in Python script)\n# Alternatively, set them manually:\n# os.environ['OPENAI_API_KEY'] = 'your-openai-api-key'\n# os.environ['HF_TOKEN'] = 'your-huggingface-token'  # optional\n\n# Load the environment\nenv = vf.load_environment(\"sv-env-network-logs\")\n\n# Evaluate a model\nresults = env.evaluate(\n    client=vf.OpenAIClient(),\n    model=\"gpt-5-mini\",\n    num_examples=100\n)\n\nprint(f\"Average reward: {results.stats['mean_reward']:.2%}\")\n```\n\n### Quick Evaluation\n\nUse the verifiers CLI for quick testing:\n\n```bash\n# First, load environment variables from .env file\nset -a && source .env && set +a\n\n# For OpenAI models (requires OPENAI_API_KEY environment variable)\nvf-eval sv-env-network-logs \\\n  --model gpt-5-mini \\\n  --num-examples 10\n\n# With custom API endpoint\nvf-eval sv-env-network-logs \\\n  --model your-model-name \\\n  --api-host-base https://your-api-endpoint.com/v1 \\\n  --api-key-var YOUR_API_KEY_ENV_VAR \\\n  --num-examples 10\n```\n\n#### Command Options\n\n- `-m, --model`: Model name to use for evaluation\n- `-b, --api-host-base`: Base URL for the API endpoint (e.g., `https://api.openai.com/v1`)\n- `-k, --api-key-var`: Name of the environment variable containing your API key\n- `--num-examples`: Number of examples to evaluate (default: 100)\n- `--verbose`: Enable verbose output for debugging\n\n### Training with Prime RL\n\nIn your Prime RL orchestrator configuration:\n\n```toml\n[environment]\nid = \"sv-env-network-logs\"\n```\n\nThen launch training:\n\n```bash\n# First, load environment variables from .env file\nset -a && source .env && set +a\n\n# Then run training\nuv run rl \\\n  --trainer.model.name \"Qwen/Qwen-7B\" \\\n  --orchestrator.environment.id \"sv-env-network-logs\" \\\n  --trainer.steps 1000\n```\n\n## Task Details\n\n### Input Format\n\nNetwork log entries with connection metadata:\n\n```text\n\"Log Entry: id.orig_h=None, id.orig_p=None, id.resp_h=None, id.resp_p=8081, proto=tcp, service=None, detailed-label=None\"\n```\n\n### Expected Output\n\nStrict JSON object:\n\n```json\n{\n  \"label\": \"Benign|Malicious|Abstain\",\n  \"confidence\": 0.0,\n  \"rationale\": \"string (optional)\"\n}\n```\n\n### Scoring\n\nThe environment uses a weighted multi-criteria rubric:\n\n- **Classification Accuracy** (1.0)\n- **Format Compliance** (0.1)\n- **Calibration Bonus** (0.2)\n- **Asymmetric Cost** (0.5, heavy penalty for false negatives)\n\nTotal reward is the weighted sum of these components.\n\n## Performance Benchmarks\n\n| Model       | Accuracy | Format | Calibration | Overall |\n| ----------- | -------- | ------ | ----------- | ------- |\n| GPT-4o-mini | 60.3%    | 100%   | 85%         | 82%     |\n\nBenchmarks on 100 examples from the IoT-23 dataset (illustrative).\n\n## Dataset\n\nThe environment uses locally-built datasets derived from public network intrusion detection datasets. Build datasets with `make data-e1` before use. Available datasets:\n\n- **iot23-train-dev-test-v1.jsonl** (N=1800): Primary dataset from IoT-23, 70/15/15 train/dev/test split\n- **cic-ids-2017-ood-v1.jsonl** (N=600): Out-of-distribution dataset from CIC-IDS-2017\n- **unsw-nb15-ood-v1.jsonl** (N=600): Out-of-distribution dataset from UNSW-NB15\n\nA synthetic fallback dataset ensures the environment works for testing even without built datasets.\n\n## Requirements\n\n- Python 3.12+\n- `verifiers>=0.1.4`\n- API key for model inference (e.g., OpenAI API key)\n- HuggingFace token (only for building datasets with `make data-e1`)\n\n## Weights & Biases Logging\n\nThis environment supports automatic Weave tracing for comprehensive experiment tracking:\n\n```python\nimport wandb\nimport weave\nimport verifiers as vf\n\n# Initialize Weave (auto-traces all Verifiers operations)\nweave.init(project=\"network-logs-security\")\n\n# Load and evaluate\nenv = vf.load_environment(\"intertwine/sv-env-network-logs\")\nresults = env.evaluate(\n    client=vf.OpenAIClient(),\n    model=\"gpt-5-mini\",\n    num_examples=100\n)\n\n# Results are automatically traced to W&B\n```\n\nConfigure Weave via environment variables:\n\n- `WEAVE_PROJECT`: Set project name (default: security-verifiers)\n- `WEAVE_DISABLED`: Set to 'true' to disable logging\n- `WANDB_API_KEY`: Your W&B API key for cloud logging\n\n## Evaluation Approach\n\n### Metrics Tracked\n\n- **Accuracy**: Correct classification rate (Malicious/Benign/Abstain)\n- **Format Compliance**: Valid JSON output adherence\n- **Calibration Score**: Confidence alignment with actual accuracy\n- **Asymmetric Cost**: False negative penalty (missing attacks is worse than false alarms)\n- **Overall Reward**: Weighted combination of all metrics\n\n### Example Evaluation Script\n\n```python\nimport verifiers as vf\nimport weave\n\n# Initialize tracking\nweave.init(project=\"security-eval\")\n\nenv = vf.load_environment(\"intertwine/sv-env-network-logs\")\n\n# Run evaluation\nresults = env.evaluate(\n    client=vf.OpenAIClient(),\n    model=\"gpt-5-mini\",\n    num_examples=500,\n    seed=42\n)\n\nprint(f\"Mean Reward: {results.stats['mean_reward']:.2%}\")\nprint(f\"Accuracy: {results.stats.get('accuracy', 0):.2%}\")\nprint(f\"Calibration: {results.stats.get('calibration', 0):.2%}\")\n```\n\n## Future Improvements\n\n- **Enhanced Dataset**: Expand beyond IoT-23 to include enterprise network traffic patterns\n- **Multi-turn Interaction**: Support for requesting additional context or log entries\n- **Explainability**: Require detailed rationale for high-stakes classifications\n- **Active Learning**: Dynamic example selection based on model uncertainty\n- **Temporal Analysis**: Support for analyzing sequences of related network events\n- **Cost Customization**: Allow environment users to specify their own false positive/negative costs\n\n## About\n\nThis environment is part of the Open Security Verifiers suite - a collection of security and alignment RL environments using Prime Intellect's Verifiers framework. Each environment provides executable, programmatic rewards for training robust security-aware AI systems.\n\n## Support\n\nFor issues or questions:\n\n- Report issues on the [Prime Intellect Environments Hub](https://app.primeintellect.ai/dashboard/environments)\n- Check the [Security Verifiers GitHub repository](https://github.com/intertwine/security-verifiers)\n- Contact the Intertwine team\n","encoding":"utf-8","truncated":false,"total_bytes":11026},"status":null}