docs(api): document MIME and PDF processor options

api-reference/endpoint/extract.mdx

@@ -37,6 +37,17 @@ Exactly one of `url`, `html`, or `markdown` must be supplied as the source.
   HTML pre-processing mode: `"normal"`, `"reader"`, or `"prune"`.
 </ParamField>
 
+<ParamField body="allowedTypes" type="array">
+  Optional non-empty MIME allowlist for URL input. Omit it to allow every supported type; `"all"` and `"*"` are not accepted.
+</ParamField>
+
+<ParamField body="processors" type="array">
+  Optional processing configuration for URL input. Omit this field to use the 25-page PDF cap. You
+  may also send `{"type":"pdf"}` and omit `maxPages`; it defaults to `25`. Set `maxPages` only to
+  override the default, using `1`–`500`, or `-1` for no page limit. PDF processing costs 1 credit
+  per page actually processed.
+</ParamField>
+
 <ParamField body="fetchConfig" type="object">
   Fetch-time options. See the [Scrape endpoint](/api-reference/endpoint/scrape#request-body) for the full field list (`mode`, `stealth`, `headers`, `cookies`, `scrolls`, `wait`, `timeout`, `country`). Ignored when `html` or `markdown` is supplied.
 </ParamField>

api-reference/endpoint/scrape.mdx

@@ -34,6 +34,17 @@ Returns markdown, HTML, links, images, summary, JSON extraction, branding, or sc
   Override auto-detected content type. Common values: `"text/html"`, `"application/pdf"`.
 </ParamField>
 
+<ParamField body="allowedTypes" type="array">
+  Optional non-empty MIME allowlist. Omit it to allow every supported type; `"all"` and `"*"` are not accepted.
+</ParamField>
+
+<ParamField body="processors" type="array">
+  Optional processing configuration. Omit this field to use the 25-page PDF cap. You may also send
+  `{"type":"pdf"}` and omit `maxPages`; it defaults to `25`. Set `maxPages` only to override the
+  default, using `1`–`500`, or `-1` for no page limit. PDF processing costs 1 credit per page
+  actually processed.
+</ParamField>
+
 <ParamField body="fetchConfig" type="object">
   Fetch-time options. All fields are optional.
 

api-reference/endpoint/search.mdx

@@ -39,6 +39,17 @@ Replaces the v1 `searchscraper` endpoint. Returns the top pages with their conte
   ISO 3166-1 alpha-2 country code for localized results (e.g. `"us"`, `"it"`).
 </ParamField>
 
+<ParamField body="allowedTypes" type="array">
+  Optional MIME allowlist for fetched results. Omit this field to allow every supported type, including `application/pdf`. There is no `"all"` or `"*"` keyword. When provided, it must be a non-empty array of exact supported MIME types, such as `["text/html", "application/pdf"]`. Rejected types are reported as failed pages and do not appear in `results`.
+</ParamField>
+
+<ParamField body="processors" type="array">
+  Optional processing configuration. Omit this field to use the 25-page PDF cap. You may also send
+  `{"type":"pdf"}` and omit `maxPages`; it defaults to `25`. Set `maxPages` only to override the
+  default, using `1`–`500`, or `-1` for no page limit. PDF processing costs 1 credit per page
+  actually processed. This does not change `allowedTypes`.
+</ParamField>
+
 <ParamField body="fetchConfig" type="object">
   Fetch-time options applied when crawling each result. See the [Scrape endpoint](/api-reference/endpoint/scrape#request-body) for the full field list.
 </ParamField>
@@ -51,7 +62,9 @@ curl -X POST https://v2-api.scrapegraphai.com/api/search \
   -H "Content-Type: application/json" \
   -d '{
     "query": "scrapegraphai pricing",
-    "numResults": 3
+    "numResults": 3,
+    "allowedTypes": ["text/html", "application/pdf"],
+    "processors": [{"type": "pdf", "maxPages": 10}]
   }'
 ```
 

docs.json

@@ -81,8 +81,7 @@
                       "services/mcp-server/cursor",
                       "services/mcp-server/claude",
                       "services/mcp-server/claude-code",
-                      "services/mcp-server/codex",
-                      "services/mcp-server/smithery"
+                      "services/mcp-server/codex"
                     ]
                   }
                 ]
@@ -321,7 +320,7 @@
           "href": "https://scrapegraphai.com/",
           "icon": "globe"
         },
-{
+        {
           "anchor": "Blog",
           "href": "https://scrapegraphai.com/blog",
           "icon": "newspaper"
@@ -389,5 +388,11 @@
       "cursor",
       "vscode"
     ]
-  }
+  },
+  "redirects": [
+    {
+      "source": "/services/mcp-server/smithery",
+      "destination": "/services/mcp-server/introduction"
+    }
+  ]
 }

sdks/javascript.mdx

@@ -240,6 +240,8 @@ if (res.status === "success") {
 | `format`          | `string`      | No       | `"markdown"` (default) or `"html"`                                  |
 | `timeRange`       | `string`      | No       | `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"` |
 | `locationGeoCode` | `string`      | No       | Two-letter country code (e.g. `"us"`)                               |
+| `allowedTypes`    | `string[]`    | No       | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are not accepted |
+| `processors`      | `object[]`    | No       | Omit for the 25-page PDF cap. `{ type: "pdf" }` also defaults to 25; set `maxPages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed |
 | `fetchConfig`     | `FetchConfig` | No       | Fetch configuration                                                 |
 
 <Accordion title="Search + extraction" icon="code">
@@ -301,7 +303,8 @@ await sgai.crawl.delete(crawlId);
 | `allowExternal`    | `boolean`        | No       | Allow crossing domains. Default: `false`                 |
 | `includePatterns`  | `string[]`       | No       | URL patterns to include                                  |
 | `excludePatterns`  | `string[]`       | No       | URL patterns to exclude                                  |
-| `contentTypes`     | `string[]`       | No       | Allowed content types                                    |
+| `allowedTypes`     | `string[]`       | No       | Non-empty MIME allowlist; omit it to allow every supported type |
+| `processors`       | `object[]`       | No       | Omit for the 25-page PDF cap. `{ type: "pdf" }` also defaults to 25; set `maxPages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed |
 | `fetchConfig`      | `FetchConfig`    | No       | Fetch configuration                                      |
 
 ### `sgai.monitor.*`

sdks/python.mdx

@@ -270,6 +270,8 @@ if res.status == "success":
 | `schema`            | `dict`        | No       | JSON Schema for structured output. Pass a Pydantic model's `model_json_schema()` to reuse a `BaseModel`. |
 | `location_geo_code` | `str`         | No       | Two-letter country code (e.g. `"us"`, `"it"`)                             |
 | `time_range`        | `str`         | No       | `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"` |
+| `allowed_types`     | `list[str]`   | No       | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are not accepted |
+| `processors`        | `list[PdfProcessor]` | No | Omit for the 25-page PDF cap. `PdfProcessor()` also defaults to 25; set `max_pages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed |
 | `fetch_config`      | `FetchConfig` | No       | Fetch configuration                                                       |
 
 ### Crawl
@@ -316,7 +318,8 @@ sgai.crawl.delete(crawl_id)
 | `allow_external`     | `bool`                    | No       | Default `False`                                          |
 | `include_patterns`   | `list[str]`               | No       | URL glob patterns to include                             |
 | `exclude_patterns`   | `list[str]`               | No       | URL glob patterns to exclude                             |
-| `content_types`      | `list[str]`               | No       | Allowed response content types                           |
+| `allowed_types`      | `list[str]`               | No       | Non-empty MIME allowlist; omit it to allow every supported type |
+| `processors`         | `list[PdfProcessor]`       | No       | Omit for the 25-page PDF cap. `PdfProcessor()` also defaults to 25; set `max_pages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed |
 | `fetch_config`       | `FetchConfig`             | No       | Fetch configuration                                      |
 
 ### Monitor

services/crawl.mdx

@@ -133,7 +133,8 @@ curl -X GET "https://v2-api.scrapegraphai.com/api/crawl/:id/pages?limit=50&curso
 | `allowExternal` / `allow_external` | bool | No | Whether to follow links to other domains. Default `false` (same-origin only). |
 | `includePatterns` / `include_patterns` | array | No | URL patterns to include (e.g. `["/blog/*"]`). |
 | `excludePatterns` / `exclude_patterns` | array | No | URL patterns to exclude (e.g. `["/admin/*"]`). |
-| `contentTypes` / `content_types` | array | No | Limit crawled pages to these MIME types, e.g. `["text/html", "application/pdf"]`. |
+| `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are invalid. |
+| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. |
 | `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). |
 
 <Note>

services/extract.mdx

@@ -79,6 +79,8 @@ curl -X POST https://v2-api.scrapegraphai.com/api/extract \
 | `prompt` | string | Yes | Natural-language description of what to extract. |
 | `schema` | object | No | JSON schema describing the desired output shape. In Python you can pass a Pydantic model via `MyModel.model_json_schema()`. |
 | `mode` | string | No | HTML processing mode: `"normal"`, `"reader"`, `"prune"`. |
+| `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist for URL input. Omit it for all supported types; `"all"` and `"*"` are invalid. |
+| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. |
 | `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). |
 
 <Note>

services/mcp-server.mdx

@@ -26,7 +26,4 @@ The MCP Server documentation has moved.
   <Card title="Codex" icon="/logo/codex.svg" href="/services/mcp-server/codex">
     Set up ScrapeGraph MCP in Codex.
   </Card>
-  <Card title="Smithery" icon="server" href="/services/mcp-server/smithery">
-    Install via the Smithery registry.
-  </Card>
 </CardGroup>

services/mcp-server/claude-code.mdx

@@ -1,47 +1,48 @@
 ---
 title: 'Claude Code'
-description: 'Configure ScrapeGraph MCP in Claude Code (remote-first)'
+description: 'Configure ScrapeGraph MCP in Claude Code'
 icon: '/logo/claude-color.svg'
 ---
 
 ## Claude Code Setup
 
-Use the remote HTTP MCP endpoint (recommended):
-
-```
-https://mcp.scrapegraphai.com/mcp
-```
-
 Add the server with the Claude Code CLI:
 
 ```bash
 claude mcp add --transport http sgai \
   https://mcp.scrapegraphai.com/mcp
 ```
 
-The default `local` scope makes the server available in the current project. To use it in every project, add `--scope user`.
+The default `local` scope makes the server available in the current project. Add `--scope user` to use it in every project.
 
-Verify the connection:
+## Authenticate
 
 ```bash
-claude mcp get sgai
+claude mcp login sgai
 ```
 
-Run `/mcp` inside Claude Code, select `sgai`, and choose **Authenticate**. In the browser window, choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account.
+Alternatively, run `/mcp` inside Claude Code, select `sgai`, and choose **Authenticate**. Either way a browser window opens: choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account.
+
+Once the browser shows **Authentication successful. Connected to sgai.**, return to Claude Code — the tools are ready.
 
-After the browser shows **Authentication successful. Connected to sgai.**, return to Claude Code to view and use the connected tools.
+### With an API key instead
 
-### Local (stdio) alternative
+```bash
+claude mcp add --transport http sgai \
+  https://mcp.scrapegraphai.com/mcp \
+  --header "Authorization: Bearer $SGAI_API_KEY"
+```
 
-If you prefer running locally:
+A request carrying an API key skips the OAuth flow, so there is no `claude mcp login` step.
+
+## Verify
 
 ```bash
-pip install scrapegraph-mcp
-claude mcp add sgai-local \
-  --env SGAI_API_KEY=YOUR_API_KEY \
-  -- python3 -m scrapegraph_mcp.server
+claude mcp get sgai
 ```
 
-<Note>
-  Use <code>--scope user</code> if you want the local server to be available in every project. On Windows, replace <code>python3</code> with the Python command available on your system.
-</Note>
+Then ask Claude Code to run the `credits` tool. A balance response confirms that authentication and tool execution both work.
+
+<Tip>
+  See the [tool reference](/services/mcp-server/introduction#available-tools) for everything the server exposes.
+</Tip>

services/mcp-server/claude.mdx

@@ -1,89 +1,70 @@
 ---
 title: 'Claude Desktop'
-description: 'Configure ScrapeGraph MCP in Claude Desktop (remote-first)'
+description: 'Configure ScrapeGraph MCP in Claude Desktop'
 icon: '/logo/claude-color.svg'
 ---
 
 ## Claude Desktop Setup
 
-Use the remote HTTP MCP endpoint with a lightweight proxy:
+Claude Desktop connects to MCP servers over stdio, so the hosted endpoint is reached through the [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) proxy.
 
-```
-https://mcp.scrapegraphai.com/mcp
-```
-
-Add to your Claude config (`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS):
+Add this to `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows):
 
 ```json
 {
   "mcpServers": {
-    "scrapegraph-mcp": {
+    "sgai": {
       "command": "npx",
       "args": [
-        "mcp-remote@0.1.25",
+        "mcp-remote@0.1.38",
         "https://mcp.scrapegraphai.com/mcp"
       ]
     }
   }
 }
 ```
 
-Restart Claude Desktop after saving the configuration. The first connection opens a browser window; choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account.
+<Note>
+  `npx` requires Node.js on your `PATH`. Pinning the version keeps the setup reproducible — drop the `@0.1.38` suffix if you would rather always take the latest proxy.
+</Note>
 
-### Local (stdio) alternative
+## Authenticate
 
-Install and run locally:
+Restart Claude Desktop after saving the configuration. The first connection starts the login and opens a browser window: choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account.
 
-```bash
-pip install scrapegraph-mcp
-export SGAI_API_KEY=your-api-key
-```
+Once the browser confirms the connection, return to Claude Desktop — the tools are ready.
 
-Config (`~/Library/Application Support/Claude/claude_desktop_config.json`):
+### With an API key instead
 
-```json
-{
-  "mcpServers": {
-    "scrapegraph-mcp-local": {
-      "command": "python3",
-      "args": ["-m", "scrapegraph_mcp.server"],
-      "env": { "SGAI_API_KEY": "YOUR_API_KEY" }
-    }
-  }
-}
-```
-
-#### Alternative: use the CLI directly
+Pass the key as a header and let the proxy forward it, which skips the OAuth flow:
 
 ```json
 {
   "mcpServers": {
-    "scrapegraph-mcp": {
-      "command": "scrapegraph-mcp",
-      "env": { "SGAI_API_KEY": "YOUR_API_KEY" }
+    "sgai": {
+      "command": "npx",
+      "args": [
+        "mcp-remote@0.1.38",
+        "https://mcp.scrapegraphai.com/mcp",
+        "--header",
+        "Authorization:${SGAI_AUTH_HEADER}"
+      ],
+      "env": {
+        "SGAI_AUTH_HEADER": "Bearer YOUR_API_KEY"
+      }
     }
   }
 }
 ```
 
-#### Windows tip
-
-If setting environment variables is problematic on Windows:
-
-```bash
-cmd /c "set SGAI_API_KEY=YOUR_API_KEY && python -m scrapegraph_mcp.server"
-```
-
 <Note>
-  - Make sure the CLI is on your <code>PATH</code> (might be in <code>~/.local/bin</code> or the venv).<br/>
-  - You can provide an absolute <code>python</code> path if needed (pyenv/venv).<br/>
-  - Use <code>python3</code> on macOS if <code>python</code> points to Python 2.
+  The header value is split across `args` and `env` on purpose. On Windows, Claude Desktop does not escape spaces inside `args`, so keeping `Authorization:${SGAI_AUTH_HEADER}` free of spaces avoids a mangled header.
 </Note>
 
-### Optional: Smithery
+## Verify
 
-```bash
-npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude
-```
+Ask Claude to run the `credits` tool. A balance response confirms that authentication and tool execution both work.
 
-Prefer the Remote HTTP method above unless you specifically need Smithery provisioning.
+<Tip>
+  See the [tool reference](/services/mcp-server/introduction#available-tools) for everything the server exposes.
+</Tip>