{"data":{"kind":"file","path":"README.md","version_id":"fs04sr2u4bg19i5ltzm1pgw8","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":1674,"modified_at":"2026-08-06T13:53:10.404000","content_hash":"a89d6a53e18ae3a3f6a0beeaaf8483fbe922e5e409eea12cd964d39c7bd2f1b0"},"entries":[],"content":"# API Caller / REST Client Environment\n\nEvaluate an LLM's ability to construct correct REST API calls from natural language specifications.\n\n## What It Tests\n\n- **HTTP method selection**: GET, POST, PUT, PATCH, DELETE from ambiguous descriptions\n- **URL construction**: path parameters, query strings, nested resources\n- **Header accuracy**: Content-Type, Authorization (Bearer, API key, Basic), custom headers\n- **Request body**: JSON payloads, form data, multipart structures\n- **Error handling**: status codes, retry logic, timeout detection\n- **Pagination**: offset/cursor/token-based patterns\n- **Authentication flows**: OAuth2 token refresh, API key rotation, basic auth\n\n## Scoring Dimensions (5 rubric functions)\n\n| Metric          | Weight | What It Measures                                    |\n|-----------------|--------|-----------------------------------------------------|\n| method_accuracy | 0.25   | Correct HTTP method chosen                          |\n| url_accuracy    | 0.20   | URL path, query params, path params correct         |\n| header_accuracy | 0.20   | Required headers present with correct values        |\n| body_accuracy   | 0.20   | Request body matches specification                  |\n| error_handling  | 0.15   | Proper status code handling and retry patterns      |\n\n## Dataset\n\n30 tasks across 3 difficulty levels:\n- **Level 0** (basic): Single endpoint, explicit method and params\n- **Level 1** (intermediate): Multi-step flows, auth headers, pagination\n- **Level 2** (advanced): OAuth refresh, rate limiting, conditional requests, webhooks\n\n## Usage\n\n```python\nimport verifiers as vf\nenv = vf.load_environment(\"somol/api-caller\")\n```\n","encoding":"utf-8","truncated":false,"total_bytes":1674},"status":null}