{"data":{"kind":"file","path":"README.md","version_id":"iraeu1vnr5wl5c4zks8depyw","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":4195,"modified_at":"2025-11-06T15:03:40.249000","content_hash":"92ad2b647b8aababb2043f8f11c690ed514b0a31fe904cc2bee2f82f3e8599f2"},"entries":[],"content":"# zapier-mcp\n\n### Overview\n- **Environment ID**: `zapier-mcp`\n- **Short description**: Evaluate AI agents' ability to use Zapier MCP to automate workflows across 8,000+ apps\n- **Tags**: zapier, mcp, integrations, automation, tool-use\n\n### Datasets\n- **Primary dataset(s)**: Built-in test dataset covering common Zapier integration patterns\n- **Source links**: N/A (synthetic test cases)\n- **Split sizes**: 5 test cases covering Gmail, Google Sheets, Slack, and multi-app workflows\n\n### Task\n- **Type**: Multi-turn tool use\n- **Parser**: Default message parser\n- **Rubric overview**: LLM judge evaluates successful task completion and tool usage; informational metric tracks whether MCP tools were called\n\n### Setup\n\nBefore running evaluations, you need to configure your Zapier MCP server:\n\n1. **Create a Zapier MCP Server**:\n   - Visit [https://mcp.zapier.com/](https://mcp.zapier.com/)\n   - Click \"+ New MCP Server\"\n   - Select \"Other\" as your MCP Client\n   - Provide a name (e.g., \"Prime Environments Test\")\n   - For transport type, select **\"Streamable HTTP\"** (SSE transport)\n   - Click \"Create MCP Server\"\n   - Copy the generated MCP server URL\n\n2. **Configure Tools in Zapier Dashboard**:\n   - In your MCP server dashboard, click the \"Configure\" tab\n   - Click \"+ Add tool\" to add integrations\n   - For each tool:\n     - Select the app (e.g., Gmail, Google Sheets, Slack)\n     - Choose the action/trigger (e.g., \"Send Email\", \"Create Row\")\n     - Connect your account\n     - Fill in required fields or allow AI to determine them\n     - Save the tool\n\n3. **Set Environment Variable**:\n   - Add to your `.env` file:\n     ```bash\n     ZAPIER_MCP_SERVER_URL=your_server_url_here\n     ```\n\n4. **Install Dependencies**:\n   ```bash\n   uv sync\n   ```\n\n### Quickstart\nRun an evaluation with default settings:\n\n```bash\nuv run vf-eval zapier-mcp\n```\n\nConfigure model and sampling:\n\n```bash\nuv run vf-eval zapier-mcp -m gpt-4.1 -n 5 -r 3\n```\n\nNotes:\n- Ensure your Zapier MCP server has the necessary tools configured (Gmail, Google Sheets, Slack)\n- The evaluation will test real integrations, so ensure connected accounts are valid\n\n### Environment Arguments\n\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `mcp_servers` | list | `ZAPIER_MCP_SERVER` | List of MCP server configurations to use |\n| `dataset` | Dataset | Built-in test cases | Custom dataset for evaluation |\n| `max_turns` | int | `10` | Maximum conversation turns per task |\n\n### Metrics\n\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` | Main scalar reward - 1.0 if task completed successfully, 0.0 otherwise |\n| `tool_usage_metric` | Informational - tracks whether MCP tools were called (1.0 if yes, 0.0 if no) |\n\n### Supported Zapier Integrations\n\nThe environment can work with any Zapier integration you configure in your MCP server. Common examples:\n\n**Communication:**\n- Gmail (send/read emails)\n- Slack (send messages, read channels)\n- Microsoft Teams (post messages)\n\n**Productivity:**\n- Google Sheets (create/read rows)\n- Notion (create/update pages)\n- Airtable (manage records)\n\n**CRM & Sales:**\n- Salesforce (manage leads/contacts)\n- HubSpot (update CRM data)\n\n**Other:**\n- 8,000+ apps available through Zapier\n\n### Example Tool Configurations\n\nWhen setting up tools in your Zapier MCP dashboard, here are recommended configurations for the default test dataset:\n\n1. **Gmail - Send Email**: Allow AI to determine recipient, subject, and body\n2. **Google Sheets - Create Row**: Specify sheet or allow AI to determine\n3. **Slack - Send Message**: Allow AI to determine channel and message\n4. **Gmail - Search**: Allow AI to determine search query\n5. **Google Sheets - Get Rows**: Allow AI to determine sheet name\n\n### Troubleshooting\n\n- **\"Tool not found\" errors**: Ensure tools are configured in your Zapier MCP dashboard\n- **Authentication errors**: Check that accounts are properly connected in Zapier\n- **Connection timeout**: Verify `ZAPIER_MCP_SERVER_URL` is correct and accessible\n- **Transport errors**: Ensure you selected \"Streamable HTTP\" (SSE) transport in Zapier dashboard\n- **URL format**: The Zapier MCP server URL should be in the format `https://mcp.zapier.com/sse/...`\n\n","encoding":"utf-8","truncated":false,"total_bytes":4195},"status":null}