Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion api-reference/endpoint/extract.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,8 @@ Exactly one of `url`, `html`, or `markdown` must be supplied as the source.
Optional processing configuration for URL input. Omit this field to use the 25-page PDF cap. You
may also send `{"type":"pdf"}` and omit `maxPages`; it defaults to `25`. Set `maxPages` only to
override the default, using `1`–`500`, or `-1` for no page limit. PDF processing costs 1 credit
per page actually processed.
per page actually processed. See [Configure PDF page
limits](/services/scrape#configure-pdf-page-limits) for examples.
</ParamField>

<ParamField body="fetchConfig" type="object">
Expand Down
3 changes: 2 additions & 1 deletion api-reference/endpoint/scrape.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,8 @@ Returns markdown, HTML, links, images, summary, JSON extraction, branding, or sc
Optional processing configuration. Omit this field to use the 25-page PDF cap. You may also send
`{"type":"pdf"}` and omit `maxPages`; it defaults to `25`. Set `maxPages` only to override the
default, using `1`–`500`, or `-1` for no page limit. PDF processing costs 1 credit per page
actually processed.
actually processed. See [Configure PDF page limits](/services/scrape#configure-pdf-page-limits)
for examples.
</ParamField>

<ParamField body="fetchConfig" type="object">
Expand Down
3 changes: 2 additions & 1 deletion api-reference/endpoint/search.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,8 @@ Replaces the v1 `searchscraper` endpoint. Returns the top pages with their conte
Optional processing configuration. Omit this field to use the 25-page PDF cap. You may also send
`{"type":"pdf"}` and omit `maxPages`; it defaults to `25`. Set `maxPages` only to override the
default, using `1`–`500`, or `-1` for no page limit. PDF processing costs 1 credit per page
actually processed. This does not change `allowedTypes`.
actually processed. See [Configure PDF page limits](/services/scrape#configure-pdf-page-limits)
for examples. This does not change `allowedTypes`.
</ParamField>

<ParamField body="fetchConfig" type="object">
Expand Down
4 changes: 2 additions & 2 deletions sdks/javascript.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,7 @@ if (res.status === "success") {
| `timeRange` | `string` | No | `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"` |
| `locationGeoCode` | `string` | No | Two-letter country code (e.g. `"us"`) |
| `allowedTypes` | `string[]` | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are not accepted |
| `processors` | `object[]` | No | Omit for the 25-page PDF cap. `{ type: "pdf" }` also defaults to 25; set `maxPages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed |
| `processors` | `object[]` | No | Omit for the 25-page PDF cap. `{ type: "pdf" }` also defaults to 25; set `maxPages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed. [Configuration examples](/services/scrape#configure-pdf-page-limits) |
| `fetchConfig` | `FetchConfig` | No | Fetch configuration |

<Accordion title="Search + extraction" icon="code">
Expand Down Expand Up @@ -304,7 +304,7 @@ await sgai.crawl.delete(crawlId);
| `includePatterns` | `string[]` | No | URL patterns to include |
| `excludePatterns` | `string[]` | No | URL patterns to exclude |
| `allowedTypes` | `string[]` | No | Non-empty MIME allowlist; omit it to allow every supported type |
| `processors` | `object[]` | No | Omit for the 25-page PDF cap. `{ type: "pdf" }` also defaults to 25; set `maxPages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed |
| `processors` | `object[]` | No | Omit for the 25-page PDF cap. `{ type: "pdf" }` also defaults to 25; set `maxPages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed. [Configuration examples](/services/scrape#configure-pdf-page-limits) |
| `fetchConfig` | `FetchConfig` | No | Fetch configuration |

### `sgai.monitor.*`
Expand Down
4 changes: 2 additions & 2 deletions sdks/python.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -271,7 +271,7 @@ if res.status == "success":
| `location_geo_code` | `str` | No | Two-letter country code (e.g. `"us"`, `"it"`) |
| `time_range` | `str` | No | `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"` |
| `allowed_types` | `list[str]` | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are not accepted |
| `processors` | `list[PdfProcessor]` | No | Omit for the 25-page PDF cap. `PdfProcessor()` also defaults to 25; set `max_pages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed |
| `processors` | `list[PdfProcessor]` | No | Omit for the 25-page PDF cap. `PdfProcessor()` also defaults to 25; set `max_pages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed. [Configuration examples](/services/scrape#configure-pdf-page-limits) |
| `fetch_config` | `FetchConfig` | No | Fetch configuration |

### Crawl
Expand Down Expand Up @@ -319,7 +319,7 @@ sgai.crawl.delete(crawl_id)
| `include_patterns` | `list[str]` | No | URL glob patterns to include |
| `exclude_patterns` | `list[str]` | No | URL glob patterns to exclude |
| `allowed_types` | `list[str]` | No | Non-empty MIME allowlist; omit it to allow every supported type |
| `processors` | `list[PdfProcessor]` | No | Omit for the 25-page PDF cap. `PdfProcessor()` also defaults to 25; set `max_pages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed |
| `processors` | `list[PdfProcessor]` | No | Omit for the 25-page PDF cap. `PdfProcessor()` also defaults to 25; set `max_pages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed. [Configuration examples](/services/scrape#configure-pdf-page-limits) |
| `fetch_config` | `FetchConfig` | No | Fetch configuration |

### Monitor
Expand Down
2 changes: 1 addition & 1 deletion services/crawl.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ curl -X GET "https://v2-api.scrapegraphai.com/api/crawl/:id/pages?limit=50&curso
| `includePatterns` / `include_patterns` | array | No | URL patterns to include (e.g. `["/blog/*"]`). |
| `excludePatterns` / `exclude_patterns` | array | No | URL patterns to exclude (e.g. `["/admin/*"]`). |
| `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are invalid. |
| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. |
| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. [Configure PDF page limits](/services/scrape#configure-pdf-page-limits). |
| `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). |

<Note>
Expand Down
2 changes: 1 addition & 1 deletion services/extract.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ curl -X POST https://v2-api.scrapegraphai.com/api/extract \
| `schema` | object | No | JSON schema describing the desired output shape. In Python you can pass a Pydantic model via `MyModel.model_json_schema()`. |
| `mode` | string | No | HTML processing mode: `"normal"`, `"reader"`, `"prune"`. |
| `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist for URL input. Omit it for all supported types; `"all"` and `"*"` are invalid. |
| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. |
| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. [Configure PDF page limits](/services/scrape#configure-pdf-page-limits). |
| `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). |

<Note>
Expand Down
51 changes: 51 additions & 0 deletions services/scrape.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,57 @@ curl -X POST https://v2-api.scrapegraphai.com/api/scrape \
Get your API key from the [dashboard](https://scrapegraphai.com/dashboard).
</Note>

## Configure PDF page limits

PDF page limits are configured inside the `processors` array, not as a top-level request field.
The API uses a 25-page limit when `processors` is omitted or when the PDF processor does not include
`maxPages`.

| Configuration | PDF pages processed |
|---|---|
| Omit `processors` | Up to 25 pages |
| `{"type": "pdf"}` | Up to 25 pages |
| `{"type": "pdf", "maxPages": 10}` | Up to 10 pages |
| `{"type": "pdf", "maxPages": -1}` | Every page |

Use an integer from `1` to `500` for a custom limit. The limit is an upper bound, not a fixed
charge: PDF processing costs 1 credit for each page actually processed. For example, a 6-page PDF
with `maxPages: 25` processes 6 pages and costs 6 credits.

<CodeGroup>

```python Python
from scrapegraph_py import PdfProcessor

res = sgai.scrape(
"https://example.com/document.pdf",
processors=[PdfProcessor(max_pages=10)],
)
```

```javascript JavaScript
const res = await sgai.scrape({
url: "https://example.com/document.pdf",
processors: [{ type: "pdf", maxPages: 10 }],
});
```

```bash cURL
curl -X POST https://v2-api.scrapegraphai.com/api/scrape \
-H "SGAI-APIKEY: $SGAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/document.pdf",
"formats": [{"type": "markdown"}],
"processors": [{"type": "pdf", "maxPages": 10}]
}'
```

</CodeGroup>

Python uses `max_pages`; JavaScript and the REST API use `maxPages`. The same `processors`
configuration is available on Scrape, Extract, Search, and Crawl.

<Accordion title="Example Response" icon="terminal">
```json
{
Expand Down
2 changes: 1 addition & 1 deletion services/search.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ curl -X POST https://v2-api.scrapegraphai.com/api/search \
| `timeRange` / `time_range` | string | No | Recency filter: `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"`. |
| `locationGeoCode` / `location_geo_code` | string | No | Two-letter ISO country code for localized results. Curated set (52): `ae`, `ar`, `at`, `au`, `be`, `br`, `ca`, `ch`, `cl`, `cn`, `co`, `cz`, `de`, `dk`, `eg`, `es`, `fi`, `fr`, `gb`, `gr`, `hk`, `hu`, `id`, `ie`, `il`, `in`, `it`, `jp`, `kr`, `mx`, `my`, `ng`, `nl`, `no`, `nz`, `pe`, `ph`, `pk`, `pl`, `pt`, `ro`, `ru`, `sa`, `se`, `sg`, `th`, `tr`, `tw`, `ua`, `us`, `vn`, `za`. |
| `allowedTypes` / `allowed_types` | array | No | Optional non-empty MIME allowlist. Omit it to allow every supported type, including PDF. |
| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. |
| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. [Configure PDF page limits](/services/scrape#configure-pdf-page-limits). |
| `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). |

<Note>
Expand Down