{"data":{"kind":"file","path":"README.md","version_id":"sbhdx16495x6bf4yvudb04hj","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":7402,"modified_at":"2025-09-18T10:54:26.843000","content_hash":"d8bd97ce973bb51cadbdad60f136ac7d1a55f84f66c54bd91b6525edbfd179a5"},"entries":[],"content":"# Web Scraper Environment\n\n### Overview\n- **Environment ID**: `web_scraper`\n- **Short description**: Advanced web scraping and API integration environment with comprehensive data extraction, parsing, and analysis tools.\n- **Tags**: web-scraping, api-integration, data-extraction, html-parsing, content-analysis\n\n### Features\nThis environment provides sophisticated web scraping and API integration capabilities:\n\n- **🌐 Web Scraping**: Fetch and parse web pages with BeautifulSoup\n- **🔌 API Integration**: Make REST API requests with full error handling\n- **📊 Data Extraction**: Extract structured data (JSON-LD, microdata, RDFa)\n- **📈 Content Analysis**: Analyze content quality and SEO metrics\n- **🔄 Retry Logic**: Robust scraping with automatic retry mechanisms\n- **🛡️ Error Handling**: Comprehensive error handling and timeout protection\n\n### Tools Available\n\n#### 1. `fetch_webpage(url: str, timeout: int = 10, headers: Optional[Dict] = None)`\nFetches a webpage and returns content with metadata.\n\n**Parameters:**\n- `url`: URL to fetch\n- `timeout`: Request timeout in seconds\n- `headers`: Optional HTTP headers\n\n**Returns:**\n- Webpage content, status code, headers, title, and response time\n\n#### 2. `parse_html_content(html_content: str, selector: str = None)`\nParses HTML content and extracts specific elements.\n\n**Parameters:**\n- `html_content`: HTML content to parse\n- `selector`: CSS selector for specific elements (optional)\n\n**Returns:**\n- Links, images, text content, meta tags, and selected elements\n\n#### 3. `extract_structured_data(html_content: str, data_type: str = \"json-ld\")`\nExtracts structured data from HTML (JSON-LD, microdata, RDFa).\n\n**Parameters:**\n- `html_content`: HTML content to parse\n- `data_type`: Type of structured data to extract\n\n**Returns:**\n- Extracted structured data in JSON format\n\n#### 4. `make_api_request(url: str, method: str = \"GET\", ...)`\nMakes API requests with full HTTP method support.\n\n**Parameters:**\n- `url`: API endpoint URL\n- `method`: HTTP method (GET, POST, PUT, DELETE)\n- `headers`: Optional HTTP headers\n- `params`: URL parameters\n- `data`: Request body data\n- `timeout`: Request timeout\n\n**Returns:**\n- API response data, status code, and headers\n\n#### 5. `analyze_content_quality(content: str, content_type: str = \"text\")`\nAnalyzes content quality and provides metrics.\n\n**Parameters:**\n- `content`: Content to analyze\n- `content_type`: Type of content (text, html, json)\n\n**Returns:**\n- Word count, readability score, quality metrics\n\n#### 6. `scrape_with_retry(url: str, max_retries: int = 3, delay: float = 1.0)`\nScrapes with retry logic for robustness.\n\n**Parameters:**\n- `url`: URL to scrape\n- `max_retries`: Maximum retry attempts\n- `delay`: Delay between retries\n\n**Returns:**\n- Final scraping result with attempt count\n\n### Quickstart\n\n#### Basic Evaluation\n```bash\nuv run vf-eval web_scraper -m gpt-4o-mini -n 5 -r 2\n```\n\n#### With Custom Parameters\n```bash\nuv run vf-eval web_scraper \\\n  -m gpt-4o-mini \\\n  -n 10 -r 3 \\\n  -a '{\"max_turns\": 10}'\n```\n\n### Example Tasks\n\nThe environment includes various web scraping challenges:\n\n#### 1. News Scraping\n- **Task**: Scrape news articles and extract key information\n- **Requirements**: Extract title, links, meta information, analyze content quality\n- **Tools Used**: `fetch_webpage`, `parse_html_content`, `analyze_content_quality`\n\n#### 2. API Integration\n- **Task**: Integrate with REST API and process response data\n- **Requirements**: Make GET request, parse JSON, extract fields, analyze structure\n- **Tools Used**: `make_api_request`, `analyze_content_quality`\n\n#### 3. Content Analysis\n- **Task**: Analyze webpage content for SEO and quality metrics\n- **Requirements**: Fetch content, parse HTML, extract text, calculate metrics\n- **Tools Used**: `fetch_webpage`, `parse_html_content`, `analyze_content_quality`\n\n#### 4. Structured Data Extraction\n- **Task**: Extract structured data from webpage\n- **Requirements**: Fetch content, look for JSON-LD, extract microdata\n- **Tools Used**: `fetch_webpage`, `extract_structured_data`\n\n### Evaluation Criteria\n\nThe environment evaluates scraping quality based on:\n\n- **Webpage Fetching** (20%): Successfully fetching web content\n- **HTML Parsing** (20%): Parsing and extracting HTML elements\n- **API Integration** (20%): Making successful API requests\n- **Content Analysis** (20%): Analyzing content quality and metrics\n- **Retry Logic** (10%): Using robust retry mechanisms\n- **Structured Data** (10%): Extracting structured data formats\n\n### Use Cases\n\n- **📰 News Aggregation**: Scrape news websites for content aggregation\n- **🛒 E-commerce Data**: Extract product information from online stores\n- **📊 Market Research**: Gather data from various web sources\n- **🔍 SEO Analysis**: Analyze website content and structure\n- **📈 Data Mining**: Extract structured data from unstructured sources\n- **🤖 Content Automation**: Automate content extraction and processing\n\n### Advanced Features\n\n#### Safe Web Scraping\n- User-Agent headers for proper identification\n- Timeout protection against slow responses\n- Error handling for network issues\n- Respectful scraping practices\n\n#### Comprehensive Data Extraction\n- JSON-LD structured data extraction\n- Microdata and RDFa support\n- CSS selector-based element extraction\n- Meta tag and SEO element detection\n\n#### Content Quality Analysis\n- Readability score calculation\n- Word and character count metrics\n- HTML structure analysis\n- SEO element detection\n\n#### Robust Error Handling\n- Automatic retry with exponential backoff\n- Network error detection and handling\n- Timeout management\n- Graceful degradation\n\n### Performance Expectations\n\n- **GPT-4o**: ~0.8-0.9 average reward (excellent scraping skills)\n- **GPT-4**: ~0.7-0.8 average reward (good scraping capabilities)\n- **Claude-3.5**: ~0.6-0.7 average reward (solid scraping skills)\n- **Smaller Models**: ~0.3-0.5 average reward (basic scraping)\n\n### Technical Details\n\n- **Max Turns**: 12 (allows for comprehensive scraping workflow)\n- **Tool Integration**: Native function calling support\n- **Safety**: Safe test URLs (httpbin.org) for evaluation\n- **Timeout**: 10-second default request timeout\n- **Retry Logic**: 3 attempts with 1-second delay\n\n### Dependencies\n\n- `requests`: HTTP library for web requests\n- `beautifulsoup4`: HTML parsing and extraction\n- `lxml`: Fast XML/HTML parser\n- `verifiers`: Core environment framework\n\n### Installation\n\n```bash\n# Install the environment\nprime env install web_scraper\n\n# Or install locally\ncd environments/web_scraper\npip install -e .\n```\n\n### Ethical Considerations\n\nThis environment is designed for educational and evaluation purposes:\n\n- Uses safe test URLs (httpbin.org) for evaluation\n- Implements respectful scraping practices\n- Includes proper User-Agent headers\n- Respects robots.txt and rate limiting\n- Designed for legitimate data extraction use cases\n\n### Contributing\n\nThis environment is designed to be extensible. You can:\n\n- Add new scraping tools and techniques\n- Implement additional data extraction methods\n- Add support for different content types\n- Create more complex scraping scenarios\n\n### Notes\n\n- Uses safe test endpoints for evaluation\n- Implements comprehensive error handling\n- Supports both simple and complex scraping tasks\n- Compatible with all major language models\n- Extensible architecture for custom tools and analysis\n","encoding":"utf-8","truncated":false,"total_bytes":7402},"status":null}