{"data":{"kind":"file","path":"README.md","version_id":"o09pn95eyrunk8kvlfpyv9zv","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3241,"modified_at":"2026-08-30T02:11:55.808000","content_hash":"c9c5738d361931d50fbd9516aabd69092a68e5c731fd7be9294ce44162956144"},"entries":[],"content":"# Accessibility Auditor\n\n**Version**: 0.0.1\n\nEvaluate an LLM's ability to audit web and application code for accessibility compliance and usability for people with disabilities.\n\n## Overview\n\nThis environment tests whether a model can identify, analyze, and recommend fixes for accessibility barriers in web/app code. It covers the full spectrum of disability categories and WCAG 2.2 success criteria.\n\n## Evaluation Dimensions\n\n### 1. WCAG Compliance Detection (weight: 0.25)\nIdentifies violations of Web Content Accessibility Guidelines across all four principles:\n- **Perceivable**: alt text, captions, color contrast, text resizing\n- **Operable**: keyboard access, timing, seizures, navigation\n- **Understandable**: readable, predictable, input assistance\n- **Robust**: parsing, assistive technology compatibility\n\n### 2. Screen Reader Compatibility (weight: 0.20)\nAnalyzes code for screen reader usability:\n- Semantic HTML structure (headings, landmarks, lists)\n- ARIA roles, states, and properties\n- Live regions and dynamic content announcements\n- Reading order and focus management\n\n### 3. Keyboard & Motor Accessibility (weight: 0.20)\nEvaluates keyboard-only and alternative input support:\n- Tab order and focus trapping\n- Custom keyboard shortcuts\n- Skip navigation links\n- Touch target sizes (minimum 44x44px)\n- Reduced motion preferences\n\n### 4. Visual & Cognitive Accessibility (weight: 0.20)\nAssesses support for low vision and cognitive disabilities:\n- Color contrast ratios (4.5:1 normal, 3:1 large text)\n- Color-only information conveyance\n- Font sizing and zoom support\n- Error identification and suggestions\n- Consistent navigation patterns\n\n### 5. Code Quality of Recommendations (weight: 0.15)\nEvaluates the technical quality of suggested fixes:\n- Correct ARIA attribute usage\n- Valid HTML semantics\n- Cross-browser/screen reader compatibility\n- Minimal code changes for maximum impact\n\n## Task Types\n\n- **Audit**: Given HTML/CSS/JS code, identify all accessibility issues\n- **Fix**: Provide corrected code that resolves identified issues\n- **Explain**: Explain why a pattern is inaccessible and its impact on users\n- **Prioritize**: Rank issues by severity and user impact\n- **Test Plan**: Generate accessibility test cases for given components\n\n## Dataset\n\n50 carefully crafted examples spanning:\n- E-commerce product pages\n- Form-heavy applications\n- Data tables and dashboards\n- Media players (video/audio)\n- Navigation menus and modals\n- Mobile-responsive layouts\n- Single-page application components\n- Email templates\n\nEach example includes realistic code snippets with deliberate accessibility violations across multiple disability categories.\n\n## Scoring\n\nRubric functions evaluate:\n1. **Issue Detection Recall**: Did the model find all real issues?\n2. **Issue Detection Precision**: Are flagged issues genuine?\n3. **WCAG Criterion Mapping**: Correct success criteria references\n4. **Severity Assessment**: Appropriate priority ranking\n5. **Fix Correctness**: Do recommended fixes actually resolve issues?\n6. **User Impact Understanding**: Demonstrates knowledge of disability experience\n\n## Usage\n\n```python\nimport verifiers as vf\n\nenv = vf.load_environment(\"lioneo/accessibility-auditor\")\n```\n\n## License\n\nMIT\n","encoding":"utf-8","truncated":false,"total_bytes":3241},"status":null}