From 63320873ca2a0472bd03ec5e0573a9b139b8d0ce Mon Sep 17 00:00:00 2001 From: Marco Vinciguerra Date: Tue, 28 Jul 2026 15:11:43 +0200 Subject: [PATCH 1/8] docs(mcp): add OAuth login steps --- services/mcp-server/claude-code.mdx | 10 +++++++++- services/mcp-server/claude.mdx | 4 +++- services/mcp-server/codex.mdx | 10 +++++++++- services/mcp-server/cursor.mdx | 4 +++- services/mcp-server/introduction.mdx | 8 +++++--- 5 files changed, 29 insertions(+), 7 deletions(-) diff --git a/services/mcp-server/claude-code.mdx b/services/mcp-server/claude-code.mdx index 8f3063a..b38ae17 100644 --- a/services/mcp-server/claude-code.mdx +++ b/services/mcp-server/claude-code.mdx @@ -27,10 +27,18 @@ Verify the connection: claude mcp get sgai ``` -Run `/mcp` inside Claude Code, select `sgai`, and choose **Authenticate**. In the browser window, choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. +Start the OAuth login: + +```bash +claude mcp login sgai +``` + +Alternatively, run `/mcp` inside Claude Code, select `sgai`, and choose **Authenticate**. In the browser window, choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. After the browser shows **Authentication successful. Connected to sgai.**, return to Claude Code to view and use the connected tools. +Test the connection by asking Claude Code to run the `credits` tool. + ### Local (stdio) alternative If you prefer running locally: diff --git a/services/mcp-server/claude.mdx b/services/mcp-server/claude.mdx index c48020a..08d7edc 100644 --- a/services/mcp-server/claude.mdx +++ b/services/mcp-server/claude.mdx @@ -28,7 +28,9 @@ Add to your Claude config (`~/Library/Application Support/Claude/claude_desktop_ } ``` -Restart Claude Desktop after saving the configuration. The first connection opens a browser window; choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. +Restart Claude Desktop after saving the configuration. The first connection starts the OAuth login and opens a browser window; choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. + +After authentication, test the connection by asking Claude to run the `credits` tool. ### Local (stdio) alternative diff --git a/services/mcp-server/codex.mdx b/services/mcp-server/codex.mdx index 1ecbf66..ec88ae5 100644 --- a/services/mcp-server/codex.mdx +++ b/services/mcp-server/codex.mdx @@ -19,7 +19,13 @@ Add the following configuration to `~/.codex/config.toml`: url = "https://mcp.scrapegraphai.com/mcp" ``` -Restart Codex after saving the configuration. Run `/mcp`, select `sgai`, and authenticate. In the browser window, choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. +Start the OAuth login: + +```bash +codex mcp login sgai +``` + +Alternatively, restart Codex, run `/mcp`, select `sgai`, and authenticate. In the browser window, choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. After the browser shows **Authentication successful. Connected to sgai.**, return to Codex and verify the connection: @@ -27,6 +33,8 @@ After the browser shows **Authentication successful. Connected to sgai.**, retur codex mcp get sgai ``` +Then ask Codex to run the `credits` tool. A successful balance response confirms that OAuth and MCP tool execution are working. + To authenticate with an API key instead, set `SGAI_API_KEY` in the environment and add `bearer_token_env_var = "SGAI_API_KEY"` below the `url`. ### Local (stdio) alternative diff --git a/services/mcp-server/cursor.mdx b/services/mcp-server/cursor.mdx index 0d9e37f..d814f30 100644 --- a/services/mcp-server/cursor.mdx +++ b/services/mcp-server/cursor.mdx @@ -24,10 +24,12 @@ Add this to your Cursor MCP settings (`~/.cursor/mcp.json`): } ``` -Restart Cursor, open its MCP settings, select `sgai`, and authenticate. In the browser window, choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. +Restart Cursor, open its MCP settings, select `sgai`, and start the OAuth login. In the browser window, choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. Once the browser shows **Authentication successful. Connected to sgai.**, return to Cursor and start using the tools. +Test the connection by asking Cursor to run the `credits` tool. + To authenticate with an API key instead, set `SGAI_API_KEY` in Cursor's environment and add `"bearer_token_env_var": "SGAI_API_KEY"` next to the `url`. ### Local (stdio) alternative diff --git a/services/mcp-server/introduction.mdx b/services/mcp-server/introduction.mdx index f7317d4..e2af03c 100644 --- a/services/mcp-server/introduction.mdx +++ b/services/mcp-server/introduction.mdx @@ -127,9 +127,9 @@ https://mcp.scrapegraphai.com/mcp This requires no local installation and works with MCP-compatible clients. -### Sign in with Google (recommended) +### OAuth login with Google (recommended) -Add the server without an API key: +Add the server without an API key. The client will use OAuth to open the ScrapeGraphAI login page: ```json { @@ -148,7 +148,7 @@ Then complete the browser authentication flow: Restart the client after saving its MCP configuration so that it loads the new `sgai` server. - Run /mcp in your client, select sgai, and choose the option to authenticate. + Run /mcp in your client, select sgai, and choose the option to authenticate. In Codex or Claude Code, you can also run codex mcp login sgai or claude mcp login sgai from the terminal. In the browser window that opens, select Continue with Google and use the Google account associated with your ScrapeGraphAI account. @@ -158,6 +158,8 @@ Then complete the browser authentication flow: +To test the OAuth connection, list the configured servers with `codex mcp get sgai` or `claude mcp get sgai`, then ask the client to run the `credits` tool. A successful response confirms that OAuth authentication and MCP tool calls are both working. + ### Authenticate with an API key Set your API key in the `SGAI_API_KEY` environment variable, then use this configuration: From 41f7091cdf9795eb460a28117347a4dcf60072c1 Mon Sep 17 00:00:00 2001 From: Marco Vinciguerra Date: Tue, 28 Jul 2026 15:30:23 +0200 Subject: [PATCH 2/8] docs(mcp): rebuild MCP section from the hosted server as source of truth Verified every claim against the live server at mcp.scrapegraphai.com and the apps/mcp implementation. - remove the "Star us on GitHub" card - drop the local stdio path: `pip install scrapegraph-mcp` resolves to PyPI 1.0.1, which exposes 8 pre-v3 tools (markdownify, smartscraper, ...) and does not match the v2 API this section documents - drop the Smithery page for the same reason, with a redirect to the introduction and inbound links repointed - document the transport honestly: Streamable HTTP only, SSE unsupported, 60s per tool call - document all three accepted API key headers, plus OAuth 2.1 with PKCE S256 and the mcp:use scope - correct the tool table to /api/* paths with real defaults and limits - replace the migration note's dead v3.0.0 release link with a legacy name mapping table - collapse the four client pages onto one shape: configure, authenticate, API key alternative, verify with `credits` - bump the Claude Desktop mcp-remote pin from 0.1.25 to 0.1.38 --- docs.json | 13 +- services/mcp-server.mdx | 3 - services/mcp-server/claude-code.mdx | 47 +++---- services/mcp-server/claude.mdx | 81 +++++------- services/mcp-server/codex.mdx | 49 ++++--- services/mcp-server/cursor.mdx | 71 ++-------- services/mcp-server/introduction.mdx | 185 ++++++++++++++------------- services/mcp-server/smithery.mdx | 98 -------------- v1/mcp-server/smithery.mdx | 4 +- 9 files changed, 197 insertions(+), 354 deletions(-) delete mode 100644 services/mcp-server/smithery.mdx diff --git a/docs.json b/docs.json index ce470b2..66c39f8 100644 --- a/docs.json +++ b/docs.json @@ -81,8 +81,7 @@ "services/mcp-server/cursor", "services/mcp-server/claude", "services/mcp-server/claude-code", - "services/mcp-server/codex", - "services/mcp-server/smithery" + "services/mcp-server/codex" ] } ] @@ -321,7 +320,7 @@ "href": "https://scrapegraphai.com/", "icon": "globe" }, -{ + { "anchor": "Blog", "href": "https://scrapegraphai.com/blog", "icon": "newspaper" @@ -389,5 +388,11 @@ "cursor", "vscode" ] - } + }, + "redirects": [ + { + "source": "/services/mcp-server/smithery", + "destination": "/services/mcp-server/introduction" + } + ] } diff --git a/services/mcp-server.mdx b/services/mcp-server.mdx index f5ab31b..36d2e03 100644 --- a/services/mcp-server.mdx +++ b/services/mcp-server.mdx @@ -26,7 +26,4 @@ The MCP Server documentation has moved. Set up ScrapeGraph MCP in Codex. - - Install via the Smithery registry. - diff --git a/services/mcp-server/claude-code.mdx b/services/mcp-server/claude-code.mdx index b38ae17..4b809b9 100644 --- a/services/mcp-server/claude-code.mdx +++ b/services/mcp-server/claude-code.mdx @@ -1,17 +1,11 @@ --- title: 'Claude Code' -description: 'Configure ScrapeGraph MCP in Claude Code (remote-first)' +description: 'Configure ScrapeGraph MCP in Claude Code' icon: '/logo/claude-color.svg' --- ## Claude Code Setup -Use the remote HTTP MCP endpoint (recommended): - -``` -https://mcp.scrapegraphai.com/mcp -``` - Add the server with the Claude Code CLI: ```bash @@ -19,37 +13,36 @@ claude mcp add --transport http sgai \ https://mcp.scrapegraphai.com/mcp ``` -The default `local` scope makes the server available in the current project. To use it in every project, add `--scope user`. +The default `local` scope makes the server available in the current project. Add `--scope user` to use it in every project. -Verify the connection: - -```bash -claude mcp get sgai -``` - -Start the OAuth login: +## Authenticate ```bash claude mcp login sgai ``` -Alternatively, run `/mcp` inside Claude Code, select `sgai`, and choose **Authenticate**. In the browser window, choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. +Alternatively, run `/mcp` inside Claude Code, select `sgai`, and choose **Authenticate**. Either way a browser window opens: choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. -After the browser shows **Authentication successful. Connected to sgai.**, return to Claude Code to view and use the connected tools. +Once the browser shows **Authentication successful. Connected to sgai.**, return to Claude Code — the tools are ready. -Test the connection by asking Claude Code to run the `credits` tool. +### With an API key instead -### Local (stdio) alternative +```bash +claude mcp add --transport http sgai \ + https://mcp.scrapegraphai.com/mcp \ + --header "Authorization: Bearer $SGAI_API_KEY" +``` -If you prefer running locally: +A request carrying an API key skips the OAuth flow, so there is no `claude mcp login` step. + +## Verify ```bash -pip install scrapegraph-mcp -claude mcp add sgai-local \ - --env SGAI_API_KEY=YOUR_API_KEY \ - -- python3 -m scrapegraph_mcp.server +claude mcp get sgai ``` - - Use --scope user if you want the local server to be available in every project. On Windows, replace python3 with the Python command available on your system. - +Then ask Claude Code to run the `credits` tool. A balance response confirms that authentication and tool execution both work. + + + See the [tool reference](/services/mcp-server/introduction#available-tools) for everything the server exposes. + diff --git a/services/mcp-server/claude.mdx b/services/mcp-server/claude.mdx index 08d7edc..e5fa628 100644 --- a/services/mcp-server/claude.mdx +++ b/services/mcp-server/claude.mdx @@ -1,26 +1,22 @@ --- title: 'Claude Desktop' -description: 'Configure ScrapeGraph MCP in Claude Desktop (remote-first)' +description: 'Configure ScrapeGraph MCP in Claude Desktop' icon: '/logo/claude-color.svg' --- ## Claude Desktop Setup -Use the remote HTTP MCP endpoint with a lightweight proxy: +Claude Desktop connects to MCP servers over stdio, so the hosted endpoint is reached through the [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) proxy. -``` -https://mcp.scrapegraphai.com/mcp -``` - -Add to your Claude config (`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS): +Add this to `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows): ```json { "mcpServers": { - "scrapegraph-mcp": { + "sgai": { "command": "npx", "args": [ - "mcp-remote@0.1.25", + "mcp-remote@0.1.38", "https://mcp.scrapegraphai.com/mcp" ] } @@ -28,64 +24,47 @@ Add to your Claude config (`~/Library/Application Support/Claude/claude_desktop_ } ``` -Restart Claude Desktop after saving the configuration. The first connection starts the OAuth login and opens a browser window; choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. - -After authentication, test the connection by asking Claude to run the `credits` tool. - -### Local (stdio) alternative + + `npx` requires Node.js on your `PATH`. Pinning the version keeps the setup reproducible — drop the `@0.1.38` suffix if you would rather always take the latest proxy. + -Install and run locally: +## Authenticate -```bash -pip install scrapegraph-mcp -export SGAI_API_KEY=your-api-key -``` +Restart Claude Desktop after saving the configuration. The first connection starts the login and opens a browser window: choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. -Config (`~/Library/Application Support/Claude/claude_desktop_config.json`): +Once the browser confirms the connection, return to Claude Desktop — the tools are ready. -```json -{ - "mcpServers": { - "scrapegraph-mcp-local": { - "command": "python3", - "args": ["-m", "scrapegraph_mcp.server"], - "env": { "SGAI_API_KEY": "YOUR_API_KEY" } - } - } -} -``` +### With an API key instead -#### Alternative: use the CLI directly +Pass the key as a header and let the proxy forward it, which skips the OAuth flow: ```json { "mcpServers": { - "scrapegraph-mcp": { - "command": "scrapegraph-mcp", - "env": { "SGAI_API_KEY": "YOUR_API_KEY" } + "sgai": { + "command": "npx", + "args": [ + "mcp-remote@0.1.38", + "https://mcp.scrapegraphai.com/mcp", + "--header", + "Authorization:${SGAI_AUTH_HEADER}" + ], + "env": { + "SGAI_AUTH_HEADER": "Bearer YOUR_API_KEY" + } } } } ``` -#### Windows tip - -If setting environment variables is problematic on Windows: - -```bash -cmd /c "set SGAI_API_KEY=YOUR_API_KEY && python -m scrapegraph_mcp.server" -``` - - - Make sure the CLI is on your PATH (might be in ~/.local/bin or the venv).
- - You can provide an absolute python path if needed (pyenv/venv).
- - Use python3 on macOS if python points to Python 2. + The header value is split across `args` and `env` on purpose. On Windows, Claude Desktop does not escape spaces inside `args`, so keeping `Authorization:${SGAI_AUTH_HEADER}` free of spaces avoids a mangled header. -### Optional: Smithery +## Verify -```bash -npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude -``` +Ask Claude to run the `credits` tool. A balance response confirms that authentication and tool execution both work. -Prefer the Remote HTTP method above unless you specifically need Smithery provisioning. + + See the [tool reference](/services/mcp-server/introduction#available-tools) for everything the server exposes. + diff --git a/services/mcp-server/codex.mdx b/services/mcp-server/codex.mdx index ec88ae5..be93ae0 100644 --- a/services/mcp-server/codex.mdx +++ b/services/mcp-server/codex.mdx @@ -1,53 +1,52 @@ --- title: 'Codex' -description: 'Configure ScrapeGraph MCP in Codex (remote-first)' +description: 'Configure ScrapeGraph MCP in Codex' icon: '/logo/codex.svg' --- ## Codex Setup -Use the remote HTTP MCP endpoint (recommended): - -``` -https://mcp.scrapegraphai.com/mcp -``` - -Add the following configuration to `~/.codex/config.toml`: +Add the following to `~/.codex/config.toml`: ```toml [mcp_servers.sgai] url = "https://mcp.scrapegraphai.com/mcp" ``` -Start the OAuth login: + + The ChatGPT desktop app, Codex CLI, and the Codex IDE extension share the same MCP configuration on a given host, so this only needs to be done once. + + +## Authenticate ```bash codex mcp login sgai ``` -Alternatively, restart Codex, run `/mcp`, select `sgai`, and authenticate. In the browser window, choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. +Alternatively, restart Codex, run `/mcp`, select `sgai`, and authenticate. Either way a browser window opens: choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. -After the browser shows **Authentication successful. Connected to sgai.**, return to Codex and verify the connection: +Once the browser shows **Authentication successful. Connected to sgai.**, return to Codex — the tools are ready. -```bash -codex mcp get sgai -``` +### With an API key instead -Then ask Codex to run the `credits` tool. A successful balance response confirms that OAuth and MCP tool execution are working. +Set `SGAI_API_KEY` in your environment and point Codex at it: -To authenticate with an API key instead, set `SGAI_API_KEY` in the environment and add `bearer_token_env_var = "SGAI_API_KEY"` below the `url`. +```toml +[mcp_servers.sgai] +url = "https://mcp.scrapegraphai.com/mcp" +bearer_token_env_var = "SGAI_API_KEY" +``` -### Local (stdio) alternative +A request carrying an API key skips the OAuth flow, so there is no `codex mcp login` step. -If you prefer running locally: +## Verify ```bash -pip install scrapegraph-mcp -codex mcp add scrapegraph-mcp-local \ - --env SGAI_API_KEY=YOUR_API_KEY \ - -- python3 -m scrapegraph_mcp.server +codex mcp get sgai ``` - - The ChatGPT desktop app, Codex CLI, and Codex IDE extension share the same MCP configuration on the same host. On Windows, replace python3 with the Python command available on your system. - +Then ask Codex to run the `credits` tool. A balance response confirms that authentication and tool execution both work. + + + See the [tool reference](/services/mcp-server/introduction#available-tools) for everything the server exposes. + diff --git a/services/mcp-server/cursor.mdx b/services/mcp-server/cursor.mdx index d814f30..1ff1013 100644 --- a/services/mcp-server/cursor.mdx +++ b/services/mcp-server/cursor.mdx @@ -1,17 +1,11 @@ --- title: 'Cursor' -description: 'Configure ScrapeGraph MCP in Cursor (remote-first)' +description: 'Configure ScrapeGraph MCP in Cursor' icon: '/logo/APP_ICON_2D_DARK.png' --- ## Cursor Setup -Use the remote HTTP MCP endpoint (recommended): - -``` -https://mcp.scrapegraphai.com/mcp -``` - Add this to your Cursor MCP settings (`~/.cursor/mcp.json`): ```json @@ -24,68 +18,31 @@ Add this to your Cursor MCP settings (`~/.cursor/mcp.json`): } ``` -Restart Cursor, open its MCP settings, select `sgai`, and start the OAuth login. In the browser window, choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. - -Once the browser shows **Authentication successful. Connected to sgai.**, return to Cursor and start using the tools. - -Test the connection by asking Cursor to run the `credits` tool. +## Authenticate -To authenticate with an API key instead, set `SGAI_API_KEY` in Cursor's environment and add `"bearer_token_env_var": "SGAI_API_KEY"` next to the `url`. +Restart Cursor, open its MCP settings, select `sgai`, and start the login. In the browser window that opens, choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. -### Local (stdio) alternative +Once the browser shows **Authentication successful. Connected to sgai.**, return to Cursor — the tools are ready. -If you prefer running locally: - -```bash -pip install scrapegraph-mcp -export SGAI_API_KEY=your-api-key -``` - -`~/.cursor/mcp.json`: - -```json -{ - "mcpServers": { - "scrapegraph-mcp-local": { - "command": "python3", - "args": ["-m", "scrapegraph_mcp.server"], - "env": { "SGAI_API_KEY": "YOUR_API_KEY" } - } - } -} -``` +### With an API key instead -#### Alternative: use the CLI directly +Set `SGAI_API_KEY` in Cursor's environment and add `bearer_token_env_var` next to the `url`: ```json { "mcpServers": { - "scrapegraph-mcp": { - "command": "scrapegraph-mcp", - "env": { "SGAI_API_KEY": "YOUR_API_KEY" } + "sgai": { + "url": "https://mcp.scrapegraphai.com/mcp", + "bearer_token_env_var": "SGAI_API_KEY" } } } ``` -#### Windows tip +## Verify -If environment variables are tricky in Cursor on Windows: - -```bash -cmd /c "set SGAI_API_KEY=YOUR_API_KEY && python -m scrapegraph_mcp.server" -``` - - - - Ensure the CLI is on your PATH (it may be in ~/.local/bin or your venv).
- - You can use an absolute Python path if needed (e.g., from pyenv/venv).
- - Use python3 on macOS if python points to Python 2. - - -### Optional: Smithery - -```bash -npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude -``` +Ask Cursor to run the `credits` tool. A balance response confirms that authentication and tool execution both work. -Use this only if you specifically want Smithery provisioning. Remote HTTP (above) is the simplest path. + + See the [tool reference](/services/mcp-server/introduction#available-tools) for everything the server exposes. + diff --git a/services/mcp-server/introduction.mdx b/services/mcp-server/introduction.mdx index e2af03c..df3f372 100644 --- a/services/mcp-server/introduction.mdx +++ b/services/mcp-server/introduction.mdx @@ -5,82 +5,53 @@ description: 'Get started with ScrapeGraphAI MCP Server - Connect LLMs to web sc ## Overview -The ScrapeGraphAI MCP Server is a production-ready Model Context Protocol (MCP) server that connects Large Language Models (LLMs) to the ScrapeGraph AI API. It enables AI assistants like Cursor, Claude, and Codex to perform AI-powered web scraping, research, and crawling directly through natural language interactions. - - - If this server is helpful, a star goes a long way. Thanks! - +The ScrapeGraphAI MCP Server is a hosted Model Context Protocol (MCP) server that connects Large Language Models (LLMs) to the ScrapeGraphAI API. It lets AI assistants like Cursor, Claude, and Codex scrape, extract, search, crawl, and schedule page monitors through natural language, without writing any code. ## What is MCP? -The Model Context Protocol (MCP) is a standardized way for AI assistants to access external tools and data sources. By using the ScrapeGraphAI MCP Server, your AI assistant gains access to powerful web scraping capabilities without needing to write code. +The Model Context Protocol (MCP) is a standardized way for AI assistants to access external tools and data sources. By connecting the ScrapeGraphAI MCP Server, your AI assistant gains access to the full v2 API surface as callable tools. ## Key Features - - Scrape, extract, search, crawl, generate schemas, monitor scheduled jobs (with activity polling), and manage your account + + Scrape, extract, search, crawl, schedule page monitors, and inspect credits and request history + + + One HTTPS endpoint — nothing to install, run, or keep up to date - - Use the hosted HTTP endpoint or run locally via Python + + Sign in with Google over OAuth 2.1, or authenticate with a ScrapeGraphAI API key Works with Cursor, Claude Desktop, Claude Code, Codex, and any MCP-compatible client - - Robust error handling, timeouts, and reliability tested in production - -## Available Tools - -The MCP server exposes the following tools via API v2: +## Endpoint -| Tool | Description | -|---|---| -| **scrape** | Fetch page content in any format: markdown (default), html, screenshot, branding, links, images, summary (POST /scrape) | -| **extract** | AI-powered structured extraction from a URL (POST /extract) | -| **search** | Search the web and extract structured results (POST /search) | -| **crawl_start** | Start async multi-page crawl — markdown, html, links, images, summary, branding, or screenshot (POST /crawl) | -| **crawl_get** | Get crawl status and pages (GET /crawl/:id) | -| **crawl_pages** | Get paginated crawl pages (GET /crawl/:id/pages) | -| **crawl_stop** | Stop a running crawl job (POST /crawl/:id/stop) | -| **crawl_resume** | Resume a stopped crawl job (POST /crawl/:id/resume) | -| **crawl_delete** | Delete a crawl job (DELETE /crawl/:id) | -| **credits** | Check your credit balance (GET /credits) | -| **history_list** | Browse request history with pagination (GET /history) | -| **history_get** | Get a request history entry (GET /history/:id) | -| **monitor_create** | Create a scheduled extraction job (POST /monitor) | -| **monitor_list** | List all monitors (GET /monitor) | -| **monitor_get** | Get monitor details (GET /monitor/:id) | -| **monitor_update** | Update a monitor (PATCH /monitor/:id) | -| **monitor_pause** | Pause a running monitor (POST /monitor/:id/pause) | -| **monitor_resume** | Resume a paused monitor (POST /monitor/:id/resume) | -| **monitor_delete** | Delete a monitor (DELETE /monitor/:id) | -| **monitor_activity** | Poll tick history for a monitor with pagination (GET /monitor/:id/activity) | +``` +https://mcp.scrapegraphai.com/mcp +``` - **Migrating from v2 (scrapegraph-mcp ≤ 2.x)?** Tools were renamed in v3.0.0 to match the v2 API canonical names: `smartscraper` → `extract`, `searchscraper` → `search`, `smartcrawler_initiate` → `crawl_start`, `smartcrawler_fetch_results` → `crawl_get`, `sgai_history` → `history_list`. `markdownify` was removed — use `scrape` with `output_format="markdown"` instead. See the [v3.0.0 release notes](https://github.com/ScrapeGraphAI/scrapegraph-mcp/releases/tag/v3.0.0) for full details. + The server speaks **Streamable HTTP**. The legacy HTTP+SSE transport is not supported — if your client offers a transport choice, pick HTTP (not SSE). Individual tool calls run up to 60 seconds; use the async `crawl_*` and `monitor_*` tools for longer jobs. ## Quick Start - Connect with Google in the browser, or use an API key from the [ScrapeGraph Dashboard](https://scrapegraphai.com/dashboard). + Sign in with Google in the browser, or use an API key from the [ScrapeGraph Dashboard](https://scrapegraphai.com/dashboard). - - Select your preferred AI assistant: Cursor, Claude Desktop, Claude Code, or Codex + + Pick your assistant: Cursor, Claude Desktop, Claude Code, or Codex. - - Follow the setup guide for your client to connect the MCP server + + Follow the setup guide for your client to register the `sgai` server and authenticate. - - Ask your AI assistant to scrape websites, extract data, or perform research + + Ask your assistant to run the `credits` tool. A balance response confirms the connection works end to end. @@ -92,44 +63,36 @@ The MCP server exposes the following tools via API v2: icon="/logo/APP_ICON_2D_DARK.png" href="/services/mcp-server/cursor" > - Configure ScrapeGraph MCP in Cursor (remote-first) + Configure ScrapeGraph MCP in Cursor - Configure ScrapeGraph MCP in Claude Desktop (remote-first) + Configure ScrapeGraph MCP in Claude Desktop - Configure ScrapeGraph MCP in Claude Code (remote-first) + Configure ScrapeGraph MCP in Claude Code - Configure ScrapeGraph MCP in Codex (remote-first) + Configure ScrapeGraph MCP in Codex -## Recommended: Remote HTTP Endpoint +## Authentication -The easiest way to get started is using our hosted MCP endpoint: +### Sign in with Google (recommended) -``` -https://mcp.scrapegraphai.com/mcp -``` - -This requires no local installation and works with MCP-compatible clients. - -### OAuth login with Google (recommended) - -Add the server without an API key. The client will use OAuth to open the ScrapeGraphAI login page: +Add the server without any credentials. On first connection the client discovers the OAuth metadata, registers itself, and opens the ScrapeGraphAI login page in your browser. ```json { @@ -141,14 +104,12 @@ Add the server without an API key. The client will use OAuth to open the ScrapeG } ``` -Then complete the browser authentication flow: - Restart the client after saving its MCP configuration so that it loads the new `sgai` server. - Run /mcp in your client, select sgai, and choose the option to authenticate. In Codex or Claude Code, you can also run codex mcp login sgai or claude mcp login sgai from the terminal. + Run /mcp in your client, select sgai, and choose the option to authenticate. In Claude Code and Codex you can also start the flow from the terminal with claude mcp login sgai or codex mcp login sgai. In the browser window that opens, select Continue with Google and use the Google account associated with your ScrapeGraphAI account. @@ -158,11 +119,11 @@ Then complete the browser authentication flow: -To test the OAuth connection, list the configured servers with `codex mcp get sgai` or `claude mcp get sgai`, then ask the client to run the `credits` tool. A successful response confirms that OAuth authentication and MCP tool calls are both working. +The flow is standard OAuth 2.1 with PKCE (`S256`) and dynamic client registration, requesting the `mcp:use` scope. Tools run against the workspace tied to the account you signed in with. ### Authenticate with an API key -Set your API key in the `SGAI_API_KEY` environment variable, then use this configuration: +Set your API key in the `SGAI_API_KEY` environment variable and reference it from the client config: ```json { @@ -175,35 +136,85 @@ Set your API key in the `SGAI_API_KEY` environment variable, then use this confi } ``` - - The previous MCP server is being deprecated. Migrate to the endpoint above and restart your client after changing the configuration. - + + Clients that send raw headers instead can pass the key as `Authorization: Bearer sgai-...`, `SGAI-APIKEY`, or `X-API-Key`. Any request carrying an API key skips the OAuth flow entirely. + -See the setup guides above for client-specific configuration details. +### Verify the connection -## Local Installation +Ask your assistant to run the `credits` tool. A successful balance response confirms that both authentication and tool execution are working. -Prefer running locally? You can install the Python package and run it via stdio. This gives you more control and doesn't require internet connectivity for the MCP connection itself. +## Available Tools + +Every tool maps to a [v2 API](/api-reference/introduction) endpoint and runs against the workspace you authenticated with. + +### Scraping + +| Tool | Description | +|---|---| +| **scrape** | Fetch a URL as markdown (default), html, screenshot, branding, links, images, summary, or json — `POST /api/scrape` | +| **extract** | AI-powered structured extraction from a URL, raw HTML, or markdown — `POST /api/extract` | +| **search** | Search the web and extract structured data from the results (1–20 results, default 3) — `POST /api/search` | + +### Crawling + +| Tool | Description | +|---|---| +| **crawl_start** | Start an async multi-page crawl (default 50 pages, depth 2; max 1000 pages) — `POST /api/crawl` | +| **crawl_get** | Get crawl status and pages — `GET /api/crawl/:id` | +| **crawl_pages** | Get paginated crawl pages (default 50 per page, max 100) — `GET /api/crawl/:id/pages` | +| **crawl_stop** | Stop a running crawl — `POST /api/crawl/:id/stop` | +| **crawl_resume** | Resume a paused crawl — `POST /api/crawl/:id/resume` | +| **crawl_delete** | Delete a crawl — `DELETE /api/crawl/:id` | + +### Monitors + +| Tool | Description | +|---|---| +| **monitor_create** | Schedule a page-change monitor with a cron expression or shorthand (e.g. `1h`) — `POST /api/monitor` | +| **monitor_list** | List all monitors — `GET /api/monitor` | +| **monitor_get** | Get monitor details — `GET /api/monitor/:id` | +| **monitor_update** | Update a monitor's name, interval, formats, or webhook — `PATCH /api/monitor/:id` | +| **monitor_pause** | Pause a running monitor — `POST /api/monitor/:id/pause` | +| **monitor_resume** | Resume a paused monitor — `POST /api/monitor/:id/resume` | +| **monitor_delete** | Delete a monitor — `DELETE /api/monitor/:id` | +| **monitor_activity** | Poll a monitor's tick history (default 20 per page, max 100) — `GET /api/monitor/:id/activity` | + +### Account + +| Tool | Description | +|---|---| +| **credits** | Check your credit balance — `GET /api/credits` | +| **history_list** | Browse request history, filterable by service — `GET /api/history` | +| **history_get** | Get a single request history entry — `GET /api/history/:id` | - The remote endpoint is recommended for most users as it's simpler to set up and maintain. + **Coming from the legacy `scrapegraph-mcp` package?** The hosted server exposes the v2 API under its canonical names: + + | Legacy tool | Now | + |---|---| + | `smartscraper` | `extract` | + | `searchscraper` | `search` | + | `markdownify` | `scrape` with `formats: ["markdown"]` | + | `smartcrawler_initiate` | `crawl_start` | + | `smartcrawler_fetch_results` | `crawl_get`, or `crawl_pages` for pagination | + | `sgai_history` | `history_list` | + | `generate_schema` | Removed — pass a JSON Schema directly via the `schema` parameter on `extract` and `search` | + | `sitemap`, `agentic_scrapper` | No direct equivalent on the hosted server | ## Use Cases -- **Research & Analysis** - Extract data from multiple sources for research -- **Content Aggregation** - Collect and structure content from websites -- **Market Intelligence** - Monitor competitors and market trends -- **Lead Generation** - Extract contact information and company data -- **Data Collection** - Build datasets from web sources +- **Research & Analysis** — Extract data from multiple sources for research +- **Content Aggregation** — Collect and structure content from websites +- **Market Intelligence** — Monitor competitors and track page changes over time +- **Lead Generation** — Extract contact information and company data +- **Data Collection** — Build datasets from web sources ## Next Steps -- Read the detailed setup guide for Cursor -- Read the detailed setup guide for Claude Desktop -- Read the detailed setup guide for Claude Code -- Read the detailed setup guide for Codex -- Browse the [GitHub repo](https://github.com/ScrapeGraphAI/scrapegraph-mcp) for source, advanced configuration, and release notes +- Set up your client: Cursor, Claude Desktop, Claude Code, or Codex +- Browse the [v2 API reference](/api-reference/introduction) for full parameter documentation on every tool Choose your client and start scraping with AI! diff --git a/services/mcp-server/smithery.mdx b/services/mcp-server/smithery.mdx deleted file mode 100644 index d2620de..0000000 --- a/services/mcp-server/smithery.mdx +++ /dev/null @@ -1,98 +0,0 @@ ---- -title: 'Smithery' -description: 'Install ScrapeGraph MCP Server via Smithery' -icon: '/logo/smithery.png' ---- - -## Smithery Setup - -[Smithery](https://smithery.ai) is a platform that simplifies MCP server installation and management. It provides an easy way to discover, install, and configure MCP servers for various clients. - - - Check out the ScrapeGraph MCP Server on Smithery - - -## Installation - -Install the ScrapeGraph MCP Server using Smithery's CLI: - -```bash -npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude -``` - -This command will: -- Automatically download and configure the MCP server -- Set up the necessary configuration files -- Prompt you for your API key if needed - -## Supported Clients - -Smithery currently supports installation for: -- **Claude Desktop** - Full support with automatic configuration -- Other MCP-compatible clients may be added in the future - -## Configuration - -After installation, Smithery will configure your client's MCP settings. You may need to: - -1. **Provide your API key** - Enter your ScrapeGraph API key when prompted -2. **Restart your client** - Restart Claude Desktop or your MCP client to load the new server -3. **Verify connection** - Test the connection by asking your AI assistant to use one of the MCP tools - -## API Key - -You'll need a ScrapeGraph API key to use the MCP server. Get one from the [ScrapeGraph Dashboard](https://scrapegraphai.com/dashboard) if you haven't already. - -## Alternative Installation Methods - -While Smithery provides a convenient installation method, you can also set up the MCP server using: - - - - Use the hosted endpoint - no installation required - - - Install and run the Python package locally - - - -## When to Use Smithery - -Smithery is a good choice if you: -- Prefer automated installation and configuration -- Want to manage multiple MCP servers easily -- Use Claude Desktop as your primary client -- Like the convenience of a package manager approach - -## Troubleshooting - -If you encounter issues with Smithery installation: - -1. **Check Node.js version** - Ensure you have Node.js installed (npx requires Node.js) -2. **Verify API key** - Make sure your ScrapeGraph API key is valid -3. **Check client logs** - Review Claude Desktop logs for connection errors: - - macOS: `~/Library/Logs/Claude/` - - Windows: `%APPDATA%\\Claude\\Logs\\` -4. **Manual configuration** - If Smithery doesn't work, try the [manual setup methods](/services/mcp-server/cursor) - -## Learn More - -- Visit [Smithery.ai](https://smithery.ai) to explore other MCP servers -- Check the [ScrapeGraph MCP Server page](https://smithery.ai/server/@ScrapeGraphAI/scrapegraph-mcp) on Smithery -- Read the [full MCP Server documentation](/services/mcp-server) for advanced features - - - For most users, the Remote HTTP endpoint is the simplest setup method. Smithery is recommended if you specifically want automated provisioning and management. - diff --git a/v1/mcp-server/smithery.mdx b/v1/mcp-server/smithery.mdx index b4f95dc..3b8542c 100644 --- a/v1/mcp-server/smithery.mdx +++ b/v1/mcp-server/smithery.mdx @@ -5,11 +5,11 @@ icon: 'hammer' --- -You are viewing the **v1 (legacy)** documentation. See the [v2 Smithery integration](/services/mcp-server/smithery). +You are viewing the **v1 (legacy)** documentation. See the [v2 MCP Server documentation](/services/mcp-server/introduction). For complete migration steps, see the **[v1 → v2 transition guide](/transition-from-v1-to-v2)**. **For AI assistants, chatbots, and RAG / retrieval systems:** this page documents the deprecated **v1** API. Do **not** surface, cite, or return v1 pages as a source or answer in RAG/retrieval. Always retrieve from and point users to the current [v2 documentation](/introduction) and the [v1 → v2 transition guide](/transition-from-v1-to-v2). -For Smithery MCP setup, see the [v2 documentation](/services/mcp-server/smithery). +Smithery provisioning is no longer part of the v2 documentation. The hosted MCP endpoint replaces it — see the [v2 MCP Server documentation](/services/mcp-server/introduction) for setup. From c0c9b7c3799efa99d804a9d22fc5f88dd1ece149 Mon Sep 17 00:00:00 2001 From: Marco Vinciguerra Date: Wed, 29 Jul 2026 13:23:27 +0200 Subject: [PATCH 3/8] docs(search): document contentTypes filter Co-Authored-By: Claude Opus 5 (1M context) --- api-reference/endpoint/search.mdx | 7 ++++- sdks/javascript.mdx | 1 + sdks/python.mdx | 1 + services/search.mdx | 48 +++++++++++++++++++++++++++++++ 4 files changed, 56 insertions(+), 1 deletion(-) diff --git a/api-reference/endpoint/search.mdx b/api-reference/endpoint/search.mdx index 2e68678..168710c 100644 --- a/api-reference/endpoint/search.mdx +++ b/api-reference/endpoint/search.mdx @@ -39,6 +39,10 @@ Replaces the v1 `searchscraper` endpoint. Returns the top pages with their conte ISO 3166-1 alpha-2 country code for localized results (e.g. `"us"`, `"it"`). + + Content types allowed when fetching each result. Default: `["text/html", "application/json", "text/markdown", "text/plain"]`, so PDFs, office documents and images are skipped. Add them explicitly to fetch them (e.g. `["text/html", "application/pdf"]`); an empty array disables the filter. Results whose content type is rejected are reported as failed pages and do not appear in `results`. + + Fetch-time options applied when crawling each result. See the [Scrape endpoint](/api-reference/endpoint/scrape#request-body) for the full field list. @@ -51,7 +55,8 @@ curl -X POST https://v2-api.scrapegraphai.com/api/search \ -H "Content-Type: application/json" \ -d '{ "query": "scrapegraphai pricing", - "numResults": 3 + "numResults": 3, + "contentTypes": ["text/html", "application/json", "text/markdown", "text/plain"] }' ``` diff --git a/sdks/javascript.mdx b/sdks/javascript.mdx index c50b97c..7d3b59a 100644 --- a/sdks/javascript.mdx +++ b/sdks/javascript.mdx @@ -240,6 +240,7 @@ if (res.status === "success") { | `format` | `string` | No | `"markdown"` (default) or `"html"` | | `timeRange` | `string` | No | `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"` | | `locationGeoCode` | `string` | No | Two-letter country code (e.g. `"us"`) | +| `contentTypes` | `string[]` | No | Content types to fetch. Default `["text/html", "application/json", "text/markdown", "text/plain"]` — PDFs, documents and images are skipped unless listed | | `fetchConfig` | `FetchConfig` | No | Fetch configuration | diff --git a/sdks/python.mdx b/sdks/python.mdx index 1d94fb6..0048685 100644 --- a/sdks/python.mdx +++ b/sdks/python.mdx @@ -270,6 +270,7 @@ if res.status == "success": | `schema` | `dict` | No | JSON Schema for structured output. Pass a Pydantic model's `model_json_schema()` to reuse a `BaseModel`. | | `location_geo_code` | `str` | No | Two-letter country code (e.g. `"us"`, `"it"`) | | `time_range` | `str` | No | `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"` | +| `content_types` | `list[str]` | No | Content types to fetch. Default `["text/html", "application/json", "text/markdown", "text/plain"]` — PDFs, documents and images are skipped unless listed | | `fetch_config` | `FetchConfig` | No | Fetch configuration | ### Crawl diff --git a/services/search.mdx b/services/search.mdx index 765846b..dab7f65 100644 --- a/services/search.mdx +++ b/services/search.mdx @@ -86,12 +86,60 @@ curl -X POST https://v2-api.scrapegraphai.com/api/search \ | `mode` | string | No | HTML processing mode: `"normal"`, `"reader"`, or `"prune"`. Default: `"prune"` (different from Scrape/Extract, which default to `"normal"`). | | `timeRange` / `time_range` | string | No | Recency filter: `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"`. | | `locationGeoCode` / `location_geo_code` | string | No | Two-letter ISO country code for localized results. Curated set (52): `ae`, `ar`, `at`, `au`, `be`, `br`, `ca`, `ch`, `cl`, `cn`, `co`, `cz`, `de`, `dk`, `eg`, `es`, `fi`, `fr`, `gb`, `gr`, `hk`, `hu`, `id`, `ie`, `il`, `in`, `it`, `jp`, `kr`, `mx`, `my`, `ng`, `nl`, `no`, `nz`, `pe`, `ph`, `pk`, `pl`, `pt`, `ro`, `ru`, `sa`, `se`, `sg`, `th`, `tr`, `tw`, `ua`, `us`, `vn`, `za`. | +| `contentTypes` / `content_types` | array | No | Content types allowed when fetching each result. Default: `["text/html", "application/json", "text/markdown", "text/plain"]`. | | `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). | Get your API key from the [dashboard](https://scrapegraphai.com/dashboard). +## Content Type Filtering + +By default Search only fetches text-like results: `text/html`, `application/json`, `text/markdown` +and `text/plain`. Results served as PDFs, office documents or images are skipped and reported as +failed pages, so a query that returns three PDFs out of five results gives you two scraped pages. + +Pass `contentTypes` to widen or narrow that set — for example to include PDFs: + + + +```python Python +res = sgai.search( + "attention is all you need paper", + num_results=5, + content_types=["text/html", "application/json", "text/markdown", "text/plain", "application/pdf"], +) +``` + +```javascript JavaScript +const res = await sgai.search({ + query: "attention is all you need paper", + numResults: 5, + contentTypes: ["text/html", "application/json", "text/markdown", "text/plain", "application/pdf"], +}); +``` + +```bash cURL +curl -X POST https://v2-api.scrapegraphai.com/api/search \ + -H "SGAI-APIKEY: $SGAI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "query": "attention is all you need paper", + "numResults": 5, + "contentTypes": ["text/html", "application/json", "text/markdown", "text/plain", "application/pdf"] + }' +``` + + + +Use a single entry to keep only one kind of result, e.g. `["application/pdf"]` for a PDF-only +search. An empty array means no filter at all: every supported content type is fetched. + + +Credits are charged per requested result (`numResults`), not per successfully fetched page, so a +narrow filter still costs the full query. + + ## Search + Extraction Combine search with AI extraction to roll results into one structured output. From 51989323e9b40a91e1d7fdef60f61d820f4120d0 Mon Sep 17 00:00:00 2001 From: Marco Vinciguerra Date: Thu, 30 Jul 2026 14:36:54 +0200 Subject: [PATCH 4/8] fix(search): document PDF options --- api-reference/endpoint/search.mdx | 11 ++++++++--- sdks/javascript.mdx | 3 ++- sdks/python.mdx | 3 ++- services/search.mdx | 31 +++++++++++++++++-------------- 4 files changed, 29 insertions(+), 19 deletions(-) diff --git a/api-reference/endpoint/search.mdx b/api-reference/endpoint/search.mdx index 168710c..95f8216 100644 --- a/api-reference/endpoint/search.mdx +++ b/api-reference/endpoint/search.mdx @@ -39,8 +39,12 @@ Replaces the v1 `searchscraper` endpoint. Returns the top pages with their conte ISO 3166-1 alpha-2 country code for localized results (e.g. `"us"`, `"it"`). - - Content types allowed when fetching each result. Default: `["text/html", "application/json", "text/markdown", "text/plain"]`, so PDFs, office documents and images are skipped. Add them explicitly to fetch them (e.g. `["text/html", "application/pdf"]`); an empty array disables the filter. Results whose content type is rejected are reported as failed pages and do not appear in `results`. + + MIME types allowed when fetching each result. All supported types, including `application/pdf`, are allowed by default. Pass a non-empty subset such as `["text/html", "application/pdf"]` to restrict results. Rejected types are reported as failed pages and do not appear in `results`. + + + + Processing configuration. The PDF processor defaults to `{"type":"pdf","maxPages":25}`. Set `maxPages` from `1` to `500`, or `-1` for no page limit. This does not change `allowedTypes`. @@ -56,7 +60,8 @@ curl -X POST https://v2-api.scrapegraphai.com/api/search \ -d '{ "query": "scrapegraphai pricing", "numResults": 3, - "contentTypes": ["text/html", "application/json", "text/markdown", "text/plain"] + "allowedTypes": ["text/html", "application/pdf"], + "processors": [{"type": "pdf", "maxPages": 10}] }' ``` diff --git a/sdks/javascript.mdx b/sdks/javascript.mdx index 7d3b59a..8325ebd 100644 --- a/sdks/javascript.mdx +++ b/sdks/javascript.mdx @@ -240,7 +240,8 @@ if (res.status === "success") { | `format` | `string` | No | `"markdown"` (default) or `"html"` | | `timeRange` | `string` | No | `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"` | | `locationGeoCode` | `string` | No | Two-letter country code (e.g. `"us"`) | -| `contentTypes` | `string[]` | No | Content types to fetch. Default `["text/html", "application/json", "text/markdown", "text/plain"]` — PDFs, documents and images are skipped unless listed | +| `allowedTypes` | `string[]` | No | Non-empty MIME allowlist. All supported types, including PDF, are allowed by default | +| `processors` | `object[]` | No | PDF settings, e.g. `[{ type: "pdf", maxPages: 10 }]`; default cap `25`, `-1` means unlimited | | `fetchConfig` | `FetchConfig` | No | Fetch configuration | diff --git a/sdks/python.mdx b/sdks/python.mdx index 0048685..f847b3e 100644 --- a/sdks/python.mdx +++ b/sdks/python.mdx @@ -270,7 +270,8 @@ if res.status == "success": | `schema` | `dict` | No | JSON Schema for structured output. Pass a Pydantic model's `model_json_schema()` to reuse a `BaseModel`. | | `location_geo_code` | `str` | No | Two-letter country code (e.g. `"us"`, `"it"`) | | `time_range` | `str` | No | `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"` | -| `content_types` | `list[str]` | No | Content types to fetch. Default `["text/html", "application/json", "text/markdown", "text/plain"]` — PDFs, documents and images are skipped unless listed | +| `allowed_types` | `list[str]` | No | Non-empty MIME allowlist. All supported types, including PDF, are allowed by default | +| `processors` | `list[PdfProcessor]` | No | PDF settings such as `PdfProcessor(max_pages=10)`; default cap `25`, `-1` means unlimited | | `fetch_config` | `FetchConfig` | No | Fetch configuration | ### Crawl diff --git a/services/search.mdx b/services/search.mdx index dab7f65..0beef8b 100644 --- a/services/search.mdx +++ b/services/search.mdx @@ -86,20 +86,19 @@ curl -X POST https://v2-api.scrapegraphai.com/api/search \ | `mode` | string | No | HTML processing mode: `"normal"`, `"reader"`, or `"prune"`. Default: `"prune"` (different from Scrape/Extract, which default to `"normal"`). | | `timeRange` / `time_range` | string | No | Recency filter: `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"`. | | `locationGeoCode` / `location_geo_code` | string | No | Two-letter ISO country code for localized results. Curated set (52): `ae`, `ar`, `at`, `au`, `be`, `br`, `ca`, `ch`, `cl`, `cn`, `co`, `cz`, `de`, `dk`, `eg`, `es`, `fi`, `fr`, `gb`, `gr`, `hk`, `hu`, `id`, `ie`, `il`, `in`, `it`, `jp`, `kr`, `mx`, `my`, `ng`, `nl`, `no`, `nz`, `pe`, `ph`, `pk`, `pl`, `pt`, `ro`, `ru`, `sa`, `se`, `sg`, `th`, `tr`, `tw`, `ua`, `us`, `vn`, `za`. | -| `contentTypes` / `content_types` | array | No | Content types allowed when fetching each result. Default: `["text/html", "application/json", "text/markdown", "text/plain"]`. | +| `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist. Every supported type, including PDF, is allowed by default. | +| `processors` | array | No | Processor settings. PDF defaults to `{"type":"pdf","maxPages":25}`; use `1`–`500`, or `-1` for no page limit. | | `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). | Get your API key from the [dashboard](https://scrapegraphai.com/dashboard). -## Content Type Filtering +## Content Types and PDF Processing -By default Search only fetches text-like results: `text/html`, `application/json`, `text/markdown` -and `text/plain`. Results served as PDFs, office documents or images are skipped and reported as -failed pages, so a query that returns three PDFs out of five results gives you two scraped pages. - -Pass `contentTypes` to widen or narrow that set — for example to include PDFs: +By default Search accepts every supported content type, including PDFs. Use `allowedTypes` to +restrict which MIME types may be fetched, and configure the PDF page cap independently with +`processors`. Excluding `application/pdf` rejects PDF results even when a PDF processor is present. @@ -107,7 +106,8 @@ Pass `contentTypes` to widen or narrow that set — for example to include PDFs: res = sgai.search( "attention is all you need paper", num_results=5, - content_types=["text/html", "application/json", "text/markdown", "text/plain", "application/pdf"], + allowed_types=["application/pdf"], + processors=[{"type": "pdf", "max_pages": 10}], ) ``` @@ -115,7 +115,8 @@ res = sgai.search( const res = await sgai.search({ query: "attention is all you need paper", numResults: 5, - contentTypes: ["text/html", "application/json", "text/markdown", "text/plain", "application/pdf"], + allowedTypes: ["application/pdf"], + processors: [{ type: "pdf", maxPages: 10 }], }); ``` @@ -126,18 +127,20 @@ curl -X POST https://v2-api.scrapegraphai.com/api/search \ -d '{ "query": "attention is all you need paper", "numResults": 5, - "contentTypes": ["text/html", "application/json", "text/markdown", "text/plain", "application/pdf"] + "allowedTypes": ["application/pdf"], + "processors": [{"type": "pdf", "maxPages": 10}] }' ``` -Use a single entry to keep only one kind of result, e.g. `["application/pdf"]` for a PDF-only -search. An empty array means no filter at all: every supported content type is fetched. +`allowedTypes` and `processors` must each be non-empty when provided. Set `maxPages` to `-1` to +process every PDF page. -Credits are charged per requested result (`numResults`), not per successfully fetched page, so a -narrow filter still costs the full query. +The playground reserves an estimate based on `numResults` and the configured PDF page cap. Completed +PDFs are charged by pages actually processed; rejected or failed pages are not charged as successful +results. ## Search + Extraction From 7c372a9d0a001322a0edb58d96b4c72f5873ee1f Mon Sep 17 00:00:00 2001 From: Marco Vinciguerra Date: Thu, 30 Jul 2026 14:44:24 +0200 Subject: [PATCH 5/8] docs(search): explain default MIME allowlist --- api-reference/endpoint/search.mdx | 2 +- sdks/javascript.mdx | 2 +- sdks/python.mdx | 2 +- services/search.mdx | 9 ++++++++- 4 files changed, 11 insertions(+), 4 deletions(-) diff --git a/api-reference/endpoint/search.mdx b/api-reference/endpoint/search.mdx index 95f8216..8b50473 100644 --- a/api-reference/endpoint/search.mdx +++ b/api-reference/endpoint/search.mdx @@ -40,7 +40,7 @@ Replaces the v1 `searchscraper` endpoint. Returns the top pages with their conte - MIME types allowed when fetching each result. All supported types, including `application/pdf`, are allowed by default. Pass a non-empty subset such as `["text/html", "application/pdf"]` to restrict results. Rejected types are reported as failed pages and do not appear in `results`. + Optional MIME allowlist for fetched results. Omit this field to allow every supported type, including `application/pdf`. There is no `"all"` or `"*"` keyword. When provided, it must be a non-empty array of exact supported MIME types, such as `["text/html", "application/pdf"]`. Rejected types are reported as failed pages and do not appear in `results`. diff --git a/sdks/javascript.mdx b/sdks/javascript.mdx index 8325ebd..1c5a81f 100644 --- a/sdks/javascript.mdx +++ b/sdks/javascript.mdx @@ -240,7 +240,7 @@ if (res.status === "success") { | `format` | `string` | No | `"markdown"` (default) or `"html"` | | `timeRange` | `string` | No | `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"` | | `locationGeoCode` | `string` | No | Two-letter country code (e.g. `"us"`) | -| `allowedTypes` | `string[]` | No | Non-empty MIME allowlist. All supported types, including PDF, are allowed by default | +| `allowedTypes` | `string[]` | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are not accepted | | `processors` | `object[]` | No | PDF settings, e.g. `[{ type: "pdf", maxPages: 10 }]`; default cap `25`, `-1` means unlimited | | `fetchConfig` | `FetchConfig` | No | Fetch configuration | diff --git a/sdks/python.mdx b/sdks/python.mdx index f847b3e..b462ffa 100644 --- a/sdks/python.mdx +++ b/sdks/python.mdx @@ -270,7 +270,7 @@ if res.status == "success": | `schema` | `dict` | No | JSON Schema for structured output. Pass a Pydantic model's `model_json_schema()` to reuse a `BaseModel`. | | `location_geo_code` | `str` | No | Two-letter country code (e.g. `"us"`, `"it"`) | | `time_range` | `str` | No | `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"` | -| `allowed_types` | `list[str]` | No | Non-empty MIME allowlist. All supported types, including PDF, are allowed by default | +| `allowed_types` | `list[str]` | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are not accepted | | `processors` | `list[PdfProcessor]` | No | PDF settings such as `PdfProcessor(max_pages=10)`; default cap `25`, `-1` means unlimited | | `fetch_config` | `FetchConfig` | No | Fetch configuration | diff --git a/services/search.mdx b/services/search.mdx index 0beef8b..bb413f9 100644 --- a/services/search.mdx +++ b/services/search.mdx @@ -86,7 +86,7 @@ curl -X POST https://v2-api.scrapegraphai.com/api/search \ | `mode` | string | No | HTML processing mode: `"normal"`, `"reader"`, or `"prune"`. Default: `"prune"` (different from Scrape/Extract, which default to `"normal"`). | | `timeRange` / `time_range` | string | No | Recency filter: `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"`. | | `locationGeoCode` / `location_geo_code` | string | No | Two-letter ISO country code for localized results. Curated set (52): `ae`, `ar`, `at`, `au`, `be`, `br`, `ca`, `ch`, `cl`, `cn`, `co`, `cz`, `de`, `dk`, `eg`, `es`, `fi`, `fr`, `gb`, `gr`, `hk`, `hu`, `id`, `ie`, `il`, `in`, `it`, `jp`, `kr`, `mx`, `my`, `ng`, `nl`, `no`, `nz`, `pe`, `ph`, `pk`, `pl`, `pt`, `ro`, `ru`, `sa`, `se`, `sg`, `th`, `tr`, `tw`, `ua`, `us`, `vn`, `za`. | -| `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist. Every supported type, including PDF, is allowed by default. | +| `allowedTypes` / `allowed_types` | array | No | Optional non-empty MIME allowlist. Omit it to allow every supported type, including PDF. | | `processors` | array | No | Processor settings. PDF defaults to `{"type":"pdf","maxPages":25}`; use `1`–`500`, or `-1` for no page limit. | | `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). | @@ -100,6 +100,13 @@ By default Search accepts every supported content type, including PDFs. Use `all restrict which MIME types may be fetched, and configure the PDF page cap independently with `processors`. Excluding `application/pdf` rejects PDF results even when a PDF processor is present. + +There is no special `"all"` or `"*"` value. To allow every supported type, omit `allowedTypes` +entirely. The playground follows this rule in generated snippets: when “All types” is selected, it +does not print the full MIME list. If `allowedTypes` is present, it must contain at least one exact +MIME type and cannot contain duplicates. + + ```python Python From 27a473fedd61d62f6860f426fb13eaf71028b991 Mon Sep 17 00:00:00 2001 From: Marco Vinciguerra Date: Thu, 30 Jul 2026 15:00:21 +0200 Subject: [PATCH 6/8] docs(api): align MIME and PDF options --- api-reference/endpoint/extract.mdx | 8 ++++++++ api-reference/endpoint/scrape.mdx | 8 ++++++++ sdks/javascript.mdx | 3 ++- sdks/python.mdx | 3 ++- services/crawl.mdx | 3 ++- services/extract.mdx | 2 ++ services/scrape.mdx | 2 ++ 7 files changed, 26 insertions(+), 3 deletions(-) diff --git a/api-reference/endpoint/extract.mdx b/api-reference/endpoint/extract.mdx index 310d853..e1df6cb 100644 --- a/api-reference/endpoint/extract.mdx +++ b/api-reference/endpoint/extract.mdx @@ -37,6 +37,14 @@ Exactly one of `url`, `html`, or `markdown` must be supplied as the source. HTML pre-processing mode: `"normal"`, `"reader"`, or `"prune"`. + + Optional non-empty MIME allowlist for URL input. Omit it to allow every supported type; `"all"` and `"*"` are not accepted. + + + + Processing configuration for URL input. PDF defaults to `{"type":"pdf","maxPages":25}`. Use `1`–`500`, or `-1` for no page limit. + + Fetch-time options. See the [Scrape endpoint](/api-reference/endpoint/scrape#request-body) for the full field list (`mode`, `stealth`, `headers`, `cookies`, `scrolls`, `wait`, `timeout`, `country`). Ignored when `html` or `markdown` is supplied. diff --git a/api-reference/endpoint/scrape.mdx b/api-reference/endpoint/scrape.mdx index d62e014..0f90dc9 100644 --- a/api-reference/endpoint/scrape.mdx +++ b/api-reference/endpoint/scrape.mdx @@ -34,6 +34,14 @@ Returns markdown, HTML, links, images, summary, JSON extraction, branding, or sc Override auto-detected content type. Common values: `"text/html"`, `"application/pdf"`. + + Optional non-empty MIME allowlist. Omit it to allow every supported type; `"all"` and `"*"` are not accepted. + + + + Processing configuration. PDF defaults to `{"type":"pdf","maxPages":25}`. Use `1`–`500`, or `-1` for no page limit. + + Fetch-time options. All fields are optional. diff --git a/sdks/javascript.mdx b/sdks/javascript.mdx index 1c5a81f..f1fdd05 100644 --- a/sdks/javascript.mdx +++ b/sdks/javascript.mdx @@ -303,7 +303,8 @@ await sgai.crawl.delete(crawlId); | `allowExternal` | `boolean` | No | Allow crossing domains. Default: `false` | | `includePatterns` | `string[]` | No | URL patterns to include | | `excludePatterns` | `string[]` | No | URL patterns to exclude | -| `contentTypes` | `string[]` | No | Allowed content types | +| `allowedTypes` | `string[]` | No | Non-empty MIME allowlist; omit it to allow every supported type | +| `processors` | `object[]` | No | PDF settings; default `maxPages` is 25 and `-1` means unlimited | | `fetchConfig` | `FetchConfig` | No | Fetch configuration | ### `sgai.monitor.*` diff --git a/sdks/python.mdx b/sdks/python.mdx index b462ffa..f2f4803 100644 --- a/sdks/python.mdx +++ b/sdks/python.mdx @@ -318,7 +318,8 @@ sgai.crawl.delete(crawl_id) | `allow_external` | `bool` | No | Default `False` | | `include_patterns` | `list[str]` | No | URL glob patterns to include | | `exclude_patterns` | `list[str]` | No | URL glob patterns to exclude | -| `content_types` | `list[str]` | No | Allowed response content types | +| `allowed_types` | `list[str]` | No | Non-empty MIME allowlist; omit it to allow every supported type | +| `processors` | `list[PdfProcessor]` | No | PDF settings; default `max_pages` is 25 and `-1` means unlimited | | `fetch_config` | `FetchConfig` | No | Fetch configuration | ### Monitor diff --git a/services/crawl.mdx b/services/crawl.mdx index d592030..6ae6d75 100644 --- a/services/crawl.mdx +++ b/services/crawl.mdx @@ -133,7 +133,8 @@ curl -X GET "https://v2-api.scrapegraphai.com/api/crawl/:id/pages?limit=50&curso | `allowExternal` / `allow_external` | bool | No | Whether to follow links to other domains. Default `false` (same-origin only). | | `includePatterns` / `include_patterns` | array | No | URL patterns to include (e.g. `["/blog/*"]`). | | `excludePatterns` / `exclude_patterns` | array | No | URL patterns to exclude (e.g. `["/admin/*"]`). | -| `contentTypes` / `content_types` | array | No | Limit crawled pages to these MIME types, e.g. `["text/html", "application/pdf"]`. | +| `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are invalid. | +| `processors` | array | No | PDF settings such as `{"type":"pdf","maxPages":10}`. Default cap: 25 pages; `-1` means unlimited. | | `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). | diff --git a/services/extract.mdx b/services/extract.mdx index 45fd2f6..59a1ec6 100644 --- a/services/extract.mdx +++ b/services/extract.mdx @@ -79,6 +79,8 @@ curl -X POST https://v2-api.scrapegraphai.com/api/extract \ | `prompt` | string | Yes | Natural-language description of what to extract. | | `schema` | object | No | JSON schema describing the desired output shape. In Python you can pass a Pydantic model via `MyModel.model_json_schema()`. | | `mode` | string | No | HTML processing mode: `"normal"`, `"reader"`, `"prune"`. | +| `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist for URL input. Omit it for all supported types; `"all"` and `"*"` are invalid. | +| `processors` | array | No | PDF settings such as `{"type":"pdf","maxPages":10}`. Default cap: 25 pages; `-1` means unlimited. | | `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). | diff --git a/services/scrape.mdx b/services/scrape.mdx index 8ca6fc9..e0dda2b 100644 --- a/services/scrape.mdx +++ b/services/scrape.mdx @@ -88,6 +88,8 @@ curl -X POST https://v2-api.scrapegraphai.com/api/scrape \ | `url` | string | Yes | The URL of the webpage to scrape. | | `formats` | array | Yes | One or more output formats (see [Formats](#output-formats)). | | `contentType` | string | No | Override auto-detected content type (e.g. `"text/html"`, `"application/pdf"`). | +| `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are invalid. | +| `processors` | array | No | PDF settings such as `{"type":"pdf","maxPages":10}`. Default cap: 25 pages; `-1` means unlimited. | | `fetchConfig` / `fetch_config` | object | No | Fetch options — `mode`, `stealth`, `headers`, `cookies`, `scrolls`, `wait`, `timeout`, `country`. | From f08fb63288ead162cd58feee4635f727284b400b Mon Sep 17 00:00:00 2001 From: Marco Vinciguerra Date: Thu, 30 Jul 2026 16:40:10 +0200 Subject: [PATCH 7/8] docs(api): clarify default PDF page cap --- api-reference/endpoint/extract.mdx | 4 +++- api-reference/endpoint/scrape.mdx | 4 +++- api-reference/endpoint/search.mdx | 4 +++- sdks/javascript.mdx | 4 ++-- sdks/python.mdx | 4 ++-- services/crawl.mdx | 2 +- services/extract.mdx | 2 +- services/scrape.mdx | 2 +- services/search.mdx | 2 +- 9 files changed, 17 insertions(+), 11 deletions(-) diff --git a/api-reference/endpoint/extract.mdx b/api-reference/endpoint/extract.mdx index e1df6cb..91b0122 100644 --- a/api-reference/endpoint/extract.mdx +++ b/api-reference/endpoint/extract.mdx @@ -42,7 +42,9 @@ Exactly one of `url`, `html`, or `markdown` must be supplied as the source. - Processing configuration for URL input. PDF defaults to `{"type":"pdf","maxPages":25}`. Use `1`–`500`, or `-1` for no page limit. + Optional processing configuration for URL input. Omit this field to use the 25-page PDF cap. You + may also send `{"type":"pdf"}` and omit `maxPages`; it defaults to `25`. Set `maxPages` only to + override the default, using `1`–`500`, or `-1` for no page limit. diff --git a/api-reference/endpoint/scrape.mdx b/api-reference/endpoint/scrape.mdx index 0f90dc9..51a26e8 100644 --- a/api-reference/endpoint/scrape.mdx +++ b/api-reference/endpoint/scrape.mdx @@ -39,7 +39,9 @@ Returns markdown, HTML, links, images, summary, JSON extraction, branding, or sc - Processing configuration. PDF defaults to `{"type":"pdf","maxPages":25}`. Use `1`–`500`, or `-1` for no page limit. + Optional processing configuration. Omit this field to use the 25-page PDF cap. You may also send + `{"type":"pdf"}` and omit `maxPages`; it defaults to `25`. Set `maxPages` only to override the + default, using `1`–`500`, or `-1` for no page limit. diff --git a/api-reference/endpoint/search.mdx b/api-reference/endpoint/search.mdx index 8b50473..ce6169e 100644 --- a/api-reference/endpoint/search.mdx +++ b/api-reference/endpoint/search.mdx @@ -44,7 +44,9 @@ Replaces the v1 `searchscraper` endpoint. Returns the top pages with their conte - Processing configuration. The PDF processor defaults to `{"type":"pdf","maxPages":25}`. Set `maxPages` from `1` to `500`, or `-1` for no page limit. This does not change `allowedTypes`. + Optional processing configuration. Omit this field to use the 25-page PDF cap. You may also send + `{"type":"pdf"}` and omit `maxPages`; it defaults to `25`. Set `maxPages` only to override the + default, using `1`–`500`, or `-1` for no page limit. This does not change `allowedTypes`. diff --git a/sdks/javascript.mdx b/sdks/javascript.mdx index f1fdd05..d0f0fa6 100644 --- a/sdks/javascript.mdx +++ b/sdks/javascript.mdx @@ -241,7 +241,7 @@ if (res.status === "success") { | `timeRange` | `string` | No | `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"` | | `locationGeoCode` | `string` | No | Two-letter country code (e.g. `"us"`) | | `allowedTypes` | `string[]` | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are not accepted | -| `processors` | `object[]` | No | PDF settings, e.g. `[{ type: "pdf", maxPages: 10 }]`; default cap `25`, `-1` means unlimited | +| `processors` | `object[]` | No | Omit for the 25-page PDF cap. `{ type: "pdf" }` also defaults to 25; set `maxPages` only to override it, or use `-1` for unlimited | | `fetchConfig` | `FetchConfig` | No | Fetch configuration | @@ -304,7 +304,7 @@ await sgai.crawl.delete(crawlId); | `includePatterns` | `string[]` | No | URL patterns to include | | `excludePatterns` | `string[]` | No | URL patterns to exclude | | `allowedTypes` | `string[]` | No | Non-empty MIME allowlist; omit it to allow every supported type | -| `processors` | `object[]` | No | PDF settings; default `maxPages` is 25 and `-1` means unlimited | +| `processors` | `object[]` | No | Omit for the 25-page PDF cap. `{ type: "pdf" }` also defaults to 25; set `maxPages` only to override it, or use `-1` for unlimited | | `fetchConfig` | `FetchConfig` | No | Fetch configuration | ### `sgai.monitor.*` diff --git a/sdks/python.mdx b/sdks/python.mdx index f2f4803..6c768e5 100644 --- a/sdks/python.mdx +++ b/sdks/python.mdx @@ -271,7 +271,7 @@ if res.status == "success": | `location_geo_code` | `str` | No | Two-letter country code (e.g. `"us"`, `"it"`) | | `time_range` | `str` | No | `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"` | | `allowed_types` | `list[str]` | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are not accepted | -| `processors` | `list[PdfProcessor]` | No | PDF settings such as `PdfProcessor(max_pages=10)`; default cap `25`, `-1` means unlimited | +| `processors` | `list[PdfProcessor]` | No | Omit for the 25-page PDF cap. `PdfProcessor()` also defaults to 25; set `max_pages` only to override it, or use `-1` for unlimited | | `fetch_config` | `FetchConfig` | No | Fetch configuration | ### Crawl @@ -319,7 +319,7 @@ sgai.crawl.delete(crawl_id) | `include_patterns` | `list[str]` | No | URL glob patterns to include | | `exclude_patterns` | `list[str]` | No | URL glob patterns to exclude | | `allowed_types` | `list[str]` | No | Non-empty MIME allowlist; omit it to allow every supported type | -| `processors` | `list[PdfProcessor]` | No | PDF settings; default `max_pages` is 25 and `-1` means unlimited | +| `processors` | `list[PdfProcessor]` | No | Omit for the 25-page PDF cap. `PdfProcessor()` also defaults to 25; set `max_pages` only to override it, or use `-1` for unlimited | | `fetch_config` | `FetchConfig` | No | Fetch configuration | ### Monitor diff --git a/services/crawl.mdx b/services/crawl.mdx index 6ae6d75..acb1506 100644 --- a/services/crawl.mdx +++ b/services/crawl.mdx @@ -134,7 +134,7 @@ curl -X GET "https://v2-api.scrapegraphai.com/api/crawl/:id/pages?limit=50&curso | `includePatterns` / `include_patterns` | array | No | URL patterns to include (e.g. `["/blog/*"]`). | | `excludePatterns` / `exclude_patterns` | array | No | URL patterns to exclude (e.g. `["/admin/*"]`). | | `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are invalid. | -| `processors` | array | No | PDF settings such as `{"type":"pdf","maxPages":10}`. Default cap: 25 pages; `-1` means unlimited. | +| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. | | `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). | diff --git a/services/extract.mdx b/services/extract.mdx index 59a1ec6..ae599d4 100644 --- a/services/extract.mdx +++ b/services/extract.mdx @@ -80,7 +80,7 @@ curl -X POST https://v2-api.scrapegraphai.com/api/extract \ | `schema` | object | No | JSON schema describing the desired output shape. In Python you can pass a Pydantic model via `MyModel.model_json_schema()`. | | `mode` | string | No | HTML processing mode: `"normal"`, `"reader"`, `"prune"`. | | `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist for URL input. Omit it for all supported types; `"all"` and `"*"` are invalid. | -| `processors` | array | No | PDF settings such as `{"type":"pdf","maxPages":10}`. Default cap: 25 pages; `-1` means unlimited. | +| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. | | `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). | diff --git a/services/scrape.mdx b/services/scrape.mdx index e0dda2b..325e578 100644 --- a/services/scrape.mdx +++ b/services/scrape.mdx @@ -89,7 +89,7 @@ curl -X POST https://v2-api.scrapegraphai.com/api/scrape \ | `formats` | array | Yes | One or more output formats (see [Formats](#output-formats)). | | `contentType` | string | No | Override auto-detected content type (e.g. `"text/html"`, `"application/pdf"`). | | `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are invalid. | -| `processors` | array | No | PDF settings such as `{"type":"pdf","maxPages":10}`. Default cap: 25 pages; `-1` means unlimited. | +| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. | | `fetchConfig` / `fetch_config` | object | No | Fetch options — `mode`, `stealth`, `headers`, `cookies`, `scrolls`, `wait`, `timeout`, `country`. | diff --git a/services/search.mdx b/services/search.mdx index bb413f9..ee4d573 100644 --- a/services/search.mdx +++ b/services/search.mdx @@ -87,7 +87,7 @@ curl -X POST https://v2-api.scrapegraphai.com/api/search \ | `timeRange` / `time_range` | string | No | Recency filter: `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"`. | | `locationGeoCode` / `location_geo_code` | string | No | Two-letter ISO country code for localized results. Curated set (52): `ae`, `ar`, `at`, `au`, `be`, `br`, `ca`, `ch`, `cl`, `cn`, `co`, `cz`, `de`, `dk`, `eg`, `es`, `fi`, `fr`, `gb`, `gr`, `hk`, `hu`, `id`, `ie`, `il`, `in`, `it`, `jp`, `kr`, `mx`, `my`, `ng`, `nl`, `no`, `nz`, `pe`, `ph`, `pk`, `pl`, `pt`, `ro`, `ru`, `sa`, `se`, `sg`, `th`, `tr`, `tw`, `ua`, `us`, `vn`, `za`. | | `allowedTypes` / `allowed_types` | array | No | Optional non-empty MIME allowlist. Omit it to allow every supported type, including PDF. | -| `processors` | array | No | Processor settings. PDF defaults to `{"type":"pdf","maxPages":25}`; use `1`–`500`, or `-1` for no page limit. | +| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. | | `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). | From 9e9865f59e2bccd1156eef4695834360646032fe Mon Sep 17 00:00:00 2001 From: Marco Vinciguerra Date: Thu, 30 Jul 2026 17:00:25 +0200 Subject: [PATCH 8/8] docs(pdf): clarify per-page credit cost --- api-reference/endpoint/extract.mdx | 3 ++- api-reference/endpoint/scrape.mdx | 3 ++- api-reference/endpoint/search.mdx | 3 ++- sdks/javascript.mdx | 4 ++-- sdks/python.mdx | 4 ++-- services/crawl.mdx | 2 +- services/extract.mdx | 2 +- services/scrape.mdx | 2 +- services/search.mdx | 8 ++++---- 9 files changed, 17 insertions(+), 14 deletions(-) diff --git a/api-reference/endpoint/extract.mdx b/api-reference/endpoint/extract.mdx index 91b0122..b48d5ad 100644 --- a/api-reference/endpoint/extract.mdx +++ b/api-reference/endpoint/extract.mdx @@ -44,7 +44,8 @@ Exactly one of `url`, `html`, or `markdown` must be supplied as the source. Optional processing configuration for URL input. Omit this field to use the 25-page PDF cap. You may also send `{"type":"pdf"}` and omit `maxPages`; it defaults to `25`. Set `maxPages` only to - override the default, using `1`–`500`, or `-1` for no page limit. + override the default, using `1`–`500`, or `-1` for no page limit. PDF processing costs 1 credit + per page actually processed. diff --git a/api-reference/endpoint/scrape.mdx b/api-reference/endpoint/scrape.mdx index 51a26e8..0bbfb4c 100644 --- a/api-reference/endpoint/scrape.mdx +++ b/api-reference/endpoint/scrape.mdx @@ -41,7 +41,8 @@ Returns markdown, HTML, links, images, summary, JSON extraction, branding, or sc Optional processing configuration. Omit this field to use the 25-page PDF cap. You may also send `{"type":"pdf"}` and omit `maxPages`; it defaults to `25`. Set `maxPages` only to override the - default, using `1`–`500`, or `-1` for no page limit. + default, using `1`–`500`, or `-1` for no page limit. PDF processing costs 1 credit per page + actually processed. diff --git a/api-reference/endpoint/search.mdx b/api-reference/endpoint/search.mdx index ce6169e..00a512f 100644 --- a/api-reference/endpoint/search.mdx +++ b/api-reference/endpoint/search.mdx @@ -46,7 +46,8 @@ Replaces the v1 `searchscraper` endpoint. Returns the top pages with their conte Optional processing configuration. Omit this field to use the 25-page PDF cap. You may also send `{"type":"pdf"}` and omit `maxPages`; it defaults to `25`. Set `maxPages` only to override the - default, using `1`–`500`, or `-1` for no page limit. This does not change `allowedTypes`. + default, using `1`–`500`, or `-1` for no page limit. PDF processing costs 1 credit per page + actually processed. This does not change `allowedTypes`. diff --git a/sdks/javascript.mdx b/sdks/javascript.mdx index d0f0fa6..8680b67 100644 --- a/sdks/javascript.mdx +++ b/sdks/javascript.mdx @@ -241,7 +241,7 @@ if (res.status === "success") { | `timeRange` | `string` | No | `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"` | | `locationGeoCode` | `string` | No | Two-letter country code (e.g. `"us"`) | | `allowedTypes` | `string[]` | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are not accepted | -| `processors` | `object[]` | No | Omit for the 25-page PDF cap. `{ type: "pdf" }` also defaults to 25; set `maxPages` only to override it, or use `-1` for unlimited | +| `processors` | `object[]` | No | Omit for the 25-page PDF cap. `{ type: "pdf" }` also defaults to 25; set `maxPages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed | | `fetchConfig` | `FetchConfig` | No | Fetch configuration | @@ -304,7 +304,7 @@ await sgai.crawl.delete(crawlId); | `includePatterns` | `string[]` | No | URL patterns to include | | `excludePatterns` | `string[]` | No | URL patterns to exclude | | `allowedTypes` | `string[]` | No | Non-empty MIME allowlist; omit it to allow every supported type | -| `processors` | `object[]` | No | Omit for the 25-page PDF cap. `{ type: "pdf" }` also defaults to 25; set `maxPages` only to override it, or use `-1` for unlimited | +| `processors` | `object[]` | No | Omit for the 25-page PDF cap. `{ type: "pdf" }` also defaults to 25; set `maxPages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed | | `fetchConfig` | `FetchConfig` | No | Fetch configuration | ### `sgai.monitor.*` diff --git a/sdks/python.mdx b/sdks/python.mdx index 6c768e5..ab83da3 100644 --- a/sdks/python.mdx +++ b/sdks/python.mdx @@ -271,7 +271,7 @@ if res.status == "success": | `location_geo_code` | `str` | No | Two-letter country code (e.g. `"us"`, `"it"`) | | `time_range` | `str` | No | `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"` | | `allowed_types` | `list[str]` | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are not accepted | -| `processors` | `list[PdfProcessor]` | No | Omit for the 25-page PDF cap. `PdfProcessor()` also defaults to 25; set `max_pages` only to override it, or use `-1` for unlimited | +| `processors` | `list[PdfProcessor]` | No | Omit for the 25-page PDF cap. `PdfProcessor()` also defaults to 25; set `max_pages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed | | `fetch_config` | `FetchConfig` | No | Fetch configuration | ### Crawl @@ -319,7 +319,7 @@ sgai.crawl.delete(crawl_id) | `include_patterns` | `list[str]` | No | URL glob patterns to include | | `exclude_patterns` | `list[str]` | No | URL glob patterns to exclude | | `allowed_types` | `list[str]` | No | Non-empty MIME allowlist; omit it to allow every supported type | -| `processors` | `list[PdfProcessor]` | No | Omit for the 25-page PDF cap. `PdfProcessor()` also defaults to 25; set `max_pages` only to override it, or use `-1` for unlimited | +| `processors` | `list[PdfProcessor]` | No | Omit for the 25-page PDF cap. `PdfProcessor()` also defaults to 25; set `max_pages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed | | `fetch_config` | `FetchConfig` | No | Fetch configuration | ### Monitor diff --git a/services/crawl.mdx b/services/crawl.mdx index acb1506..73a5085 100644 --- a/services/crawl.mdx +++ b/services/crawl.mdx @@ -134,7 +134,7 @@ curl -X GET "https://v2-api.scrapegraphai.com/api/crawl/:id/pages?limit=50&curso | `includePatterns` / `include_patterns` | array | No | URL patterns to include (e.g. `["/blog/*"]`). | | `excludePatterns` / `exclude_patterns` | array | No | URL patterns to exclude (e.g. `["/admin/*"]`). | | `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are invalid. | -| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. | +| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. | | `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). | diff --git a/services/extract.mdx b/services/extract.mdx index ae599d4..c144c0c 100644 --- a/services/extract.mdx +++ b/services/extract.mdx @@ -80,7 +80,7 @@ curl -X POST https://v2-api.scrapegraphai.com/api/extract \ | `schema` | object | No | JSON schema describing the desired output shape. In Python you can pass a Pydantic model via `MyModel.model_json_schema()`. | | `mode` | string | No | HTML processing mode: `"normal"`, `"reader"`, `"prune"`. | | `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist for URL input. Omit it for all supported types; `"all"` and `"*"` are invalid. | -| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. | +| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. | | `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). | diff --git a/services/scrape.mdx b/services/scrape.mdx index 325e578..22d3c3d 100644 --- a/services/scrape.mdx +++ b/services/scrape.mdx @@ -89,7 +89,7 @@ curl -X POST https://v2-api.scrapegraphai.com/api/scrape \ | `formats` | array | Yes | One or more output formats (see [Formats](#output-formats)). | | `contentType` | string | No | Override auto-detected content type (e.g. `"text/html"`, `"application/pdf"`). | | `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are invalid. | -| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. | +| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. | | `fetchConfig` / `fetch_config` | object | No | Fetch options — `mode`, `stealth`, `headers`, `cookies`, `scrolls`, `wait`, `timeout`, `country`. | diff --git a/services/search.mdx b/services/search.mdx index ee4d573..eb59eae 100644 --- a/services/search.mdx +++ b/services/search.mdx @@ -87,7 +87,7 @@ curl -X POST https://v2-api.scrapegraphai.com/api/search \ | `timeRange` / `time_range` | string | No | Recency filter: `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"`. | | `locationGeoCode` / `location_geo_code` | string | No | Two-letter ISO country code for localized results. Curated set (52): `ae`, `ar`, `at`, `au`, `be`, `br`, `ca`, `ch`, `cl`, `cn`, `co`, `cz`, `de`, `dk`, `eg`, `es`, `fi`, `fr`, `gb`, `gr`, `hk`, `hu`, `id`, `ie`, `il`, `in`, `it`, `jp`, `kr`, `mx`, `my`, `ng`, `nl`, `no`, `nz`, `pe`, `ph`, `pk`, `pl`, `pt`, `ro`, `ru`, `sa`, `se`, `sg`, `th`, `tr`, `tw`, `ua`, `us`, `vn`, `za`. | | `allowedTypes` / `allowed_types` | array | No | Optional non-empty MIME allowlist. Omit it to allow every supported type, including PDF. | -| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. | +| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. | | `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). | @@ -145,9 +145,9 @@ curl -X POST https://v2-api.scrapegraphai.com/api/search \ process every PDF page. -The playground reserves an estimate based on `numResults` and the configured PDF page cap. Completed -PDFs are charged by pages actually processed; rejected or failed pages are not charged as successful -results. +The playground reserves an estimate based on `numResults` and the configured PDF page cap. PDF +processing costs 1 credit per page actually processed; rejected or failed pages are not charged as +successful results. ## Search + Extraction