diff --git a/.cursor-plugin/marketplace.json b/.cursor-plugin/marketplace.json index c828e0e..1ae17f1 100644 --- a/.cursor-plugin/marketplace.json +++ b/.cursor-plugin/marketplace.json @@ -6,7 +6,7 @@ }, "metadata": { "description": "Web data plugins by Decodo: scraping, SERP, and e-commerce extraction", - "version": "0.2.2" + "version": "0.3.0" }, "plugins": [ { diff --git a/plugins/decodo/.cursor-plugin/plugin.json b/plugins/decodo/.cursor-plugin/plugin.json index ddbe6c6..944b681 100644 --- a/plugins/decodo/.cursor-plugin/plugin.json +++ b/plugins/decodo/.cursor-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "decodo", "displayName": "Decodo", - "version": "0.2.2", + "version": "0.3.0", "description": "Web scraping, SERP, and e-commerce data for AI agents. Handles JavaScript rendering, anti-bot/CAPTCHA, proxy rotation, and geo-targeting — through the Decodo CLI or the hosted Decodo MCP server.", "author": { "name": "Decodo", @@ -38,12 +38,12 @@ "DECODO_AUTH_TOKEN": { "type": "string", "title": "Decodo API key or token", - "description": "Web Data API key (or, on older plans, basic auth token) from https://dashboard.decodo.com/web-data/playground (free account = up to 2K requests, no card). The same credential authenticates the MCP server, the CLI, and the raw API." + "description": "Web Data API key (or, on older plans, basic auth token) from https://dashboard.decodo.com/web-data/playground (free account = up to 10K requests, no card). The same credential authenticates the MCP server, the CLI, and the raw API. Search, map and crawl need an API key." }, "DECODO_TOOLSETS": { "type": "string", "title": "MCP toolsets (optional)", - "description": "Which Decodo MCP toolsets to enable, comma-separated. Defaults to \"web,search\". Available: web, search, ecommerce, social_media, ai. The full server exposes ~30 tools, so enable only what you need — every enabled tool's schema is resident in each request. The decodo CLI covers all targets regardless of this setting." + "description": "Which Decodo MCP toolsets to enable, comma-separated. Defaults to \"web,search\". Available: web, search, ecommerce, social_media, ai. Enable only what you need — every enabled tool's schema is resident in each request. The decodo CLI covers all targets regardless of this setting." } }, "required": [ diff --git a/plugins/decodo/README.md b/plugins/decodo/README.md index b0c9a7a..e4ecb08 100644 --- a/plugins/decodo/README.md +++ b/plugins/decodo/README.md @@ -20,10 +20,11 @@ data instead of a 403. | Install | none | none (`npx`) or one installer command | | Needs a shell | no | yes | | Coverage | the toolsets you enable | every target, always | -| Best for | plain scrape and search | retail, social, screenshots to file, batches, `jq` | +| Best for | plain scrape, search and map | crawls, retail, social, screenshots to file, batches, `jq` | -Both ship enabled. The skill routes plain scrape/search through MCP and everything else through -the CLI, so you don't have to think about it. +Both ship enabled. The skill routes plain scrape, search and map through MCP and everything else +through the CLI, so you don't have to think about it. Crawls go through the CLI when a shell is +available, because it writes pages to a file instead of into the agent's context. The MCP server defaults to the `web,search` toolsets. Widen it with `DECODO_TOOLSETS` (`web`, `search`, `ecommerce`, `social_media`, `ai`) only if you want those targets as direct tool calls @@ -32,8 +33,8 @@ The MCP server defaults to the `web,search` toolsets. Widen it with `DECODO_TOOL ## Setup **1. Get an API key.** Grab a Web Data API key from the -[Decodo Playground](https://dashboard.decodo.com/web-data/playground). A free account includes 2,000 -requests, no card required. +[Decodo Playground](https://dashboard.decodo.com/web-data/playground). A free account includes up to +10,000 requests, no card required. **2. Set `DECODO_AUTH_TOKEN`.** Cursor prompts for it when you install the plugin, or set it later under Settings → Plugins → Decodo. That one value authenticates the MCP server and the @@ -44,7 +45,8 @@ export DECODO_AUTH_TOKEN='' # session-scoped npx -y @decodo/cli setup --token '' # CLI only, saved to ~/.config/decodo ``` -The value can be a Web Data API key or, on older plans, a basic auth token. The plugin's MCP header +The value can be a Web Data API key or, on older plans, a basic auth token. Search, map and crawl +need an API key; a basic token works for everything else. The plugin's MCP header is `Basic ${DECODO_AUTH_TOKEN}` either way: the hosted server detects the credential type from the value, not from the header scheme. diff --git a/plugins/decodo/commands/decodo-setup.md b/plugins/decodo/commands/decodo-setup.md index 98d8585..8a109c0 100644 --- a/plugins/decodo/commands/decodo-setup.md +++ b/plugins/decodo/commands/decodo-setup.md @@ -26,8 +26,8 @@ Report the result as a two-line status, then continue only for the parts that ar ## 2. Get an API key or token (needed for either lane) -Both lanes use the same Web Data API key. Older plans only have a basic auth token, which works the same way. Point the user at - — a free account gives ~2K requests with no card. +Both lanes use the same Web Data API key. Older plans only have a basic auth token, which works the same way except for search, map and crawl, which need an API key. Point the user at + — a free account gives up to 10K requests with no card. **Never read, `cat`, echo, or log the token**, and never write it into a file the user tracks in git. If a token is already configured somewhere, do not extract it to reuse it elsewhere — ask diff --git a/plugins/decodo/commands/decodo-status.md b/plugins/decodo/commands/decodo-status.md index 55bce9c..6fd27ef 100644 --- a/plugins/decodo/commands/decodo-status.md +++ b/plugins/decodo/commands/decodo-status.md @@ -12,8 +12,8 @@ Diagnose, do not fix. Report what is true, then name the single next step if som Check your own tool list for Decodo tools. Report: - **Tools present?** yes / no -- **Which ones**, and therefore which toolsets are enabled (`scrape_as_markdown`/`screenshot` → - `web`; `google_search`/`bing_search` → `search`; `amazon_*`/`walmart_*`/`target_*` → +- **Which ones**, and therefore which toolsets are enabled (`scrape_as_markdown`/`screenshot`/`map`/`crawl_*` → + `web`; `search`/`google_search` → `search`; `amazon_*`/`walmart_*`/`target_*` → `ecommerce`; `reddit_*`/`tiktok_*`/`youtube_*` → `social_media`; `chatgpt`/`perplexity`/ `google_ai_mode` → `ai`). @@ -46,6 +46,8 @@ Report version and auth source. Two things to state plainly rather than paper ov token the API would reject. Report it as "a token is configured", never as "authenticated". - **Version below 1.0.0** → flag it. 0.x stored config in a different directory and has no migration, so an upgrade looks like lost auth. +- **Version below 2.0.0** → flag it. It has no `map` or `crawl`, and its `search` flags differ + from the skill's; `npx -y @decodo/cli@latest` or reinstalling gets the current version. Exit 3 means no usable credential (`No API key or auth token found.`) or a rejected one (`Invalid credentials` for an API key, `Incorrect username or password` for a basic auth token). @@ -57,9 +59,9 @@ Give the user a short table — surface, state, next step — and nothing else: | Surface | State | |---|---| | MCP | tools present, toolsets `web,search` | -| CLI | v1.0.2, token configured (unvalidated) | +| CLI | v3.0.0, API key configured (unvalidated) | -If both lanes are live, say so and note that plain scrape/search will go through MCP while -everything else goes through the CLI. If neither is, point at `/decodo-setup`. +If both lanes are live, say so and note that plain scrape, search and map will go through MCP while +crawls and everything else go through the CLI. If neither is, point at `/decodo-setup`. Never print, echo, or partially reveal the token, even masked output copied from `whoami`. diff --git a/plugins/decodo/rules/install.mdc b/plugins/decodo/rules/install.mdc index 235756e..abebdb2 100644 --- a/plugins/decodo/rules/install.mdc +++ b/plugins/decodo/rules/install.mdc @@ -32,7 +32,7 @@ writing to it will not help. Tell the user: An exported `DECODO_AUTH_TOKEN` also resolves and takes precedence, but Cursor only inherits environment variables when it is launched from a shell, so the Settings route is the reliable -one. API keys come from (free account = up to 2K +one. API keys come from (free account = up to 10K requests, no card). Reloading is required — the header is resolved when the server connects, so a token set after @@ -79,7 +79,7 @@ This section is the **CLI's** token store, which is separate from the MCP plugin setting one does not affect the other. The user needs a Web Data API key (or, on older plans, a basic auth token) from - (free account = up to 2K requests, no card). + (free account = up to 10K requests, no card). ```bash export DECODO_AUTH_TOKEN='' # preferred: no interaction, session-scoped @@ -127,6 +127,9 @@ no "401", "unauthorized", or "auth required". The messages mean different things - `Invalid credentials` — an API key *is* configured, and the API rejected it (expired, wrong, or truncated). - `Incorrect username or password` — a basic auth token *is* configured, and the API rejected it. +- Any exit 3 from `search`, `map` or `crawl` while `decodo whoami` shows `token:` (not + `api key:`) — those three commands need an API key, and a basic token is never accepted there. + Ask the user for an API key; `scrape` and target commands keep working with the token. ```bash decodo whoami # shows the CONFIGURED source + masked token — does NOT validate it diff --git a/plugins/decodo/skills/decodo/SKILL.md b/plugins/decodo/skills/decodo/SKILL.md index 8d1591d..ad5bb0d 100644 --- a/plugins/decodo/skills/decodo/SKILL.md +++ b/plugins/decodo/skills/decodo/SKILL.md @@ -1,13 +1,14 @@ --- name: decodo description: >- - Scrape websites and extract structured web data with Decodo — search Google/Bing, - pull product data from Amazon, Walmart, Target, and collect posts from Reddit, TikTok, - YouTube and more. Decodo handles JavaScript rendering, anti-bot/CAPTCHA, proxy rotation, - and geo-targeting (125M+ IPs, 195+ locations). Reach for this skill whenever a plain - fetch is blocked or returns junk, when a page needs JS to render, or when the user wants - search results, e-commerce data, SERP data, or social media data — instead of curl, - requests, BeautifulSoup, Playwright, or Puppeteer. + Scrape websites and extract structured web data with Decodo — search the web, map a + site's URLs, crawl a whole site or section, pull product data from Amazon, Walmart, Target, + and collect posts from Reddit, TikTok, YouTube and more. Decodo handles JavaScript rendering, + anti-bot/CAPTCHA, proxy rotation, and geo-targeting (125M+ IPs, 195+ locations). Reach for + this skill whenever a plain fetch is blocked or returns junk, when a page needs JS to render, + or when the user wants search results, a list of a site's pages, many pages of one site, + e-commerce data, SERP data, or social media data — instead of curl, requests, BeautifulSoup, + Playwright, Puppeteer, or a hand-written crawler. license: MIT --- @@ -26,6 +27,7 @@ Use Decodo instead of `curl` / `requests` / `BeautifulSoup` / Playwright when th - Fetching a page that is JS-heavy, geo-restricted, rate-limited, or behind anti-bot/CAPTCHA. - A plain fetch returned a 403/429/empty body/CAPTCHA challenge. - Web search results (Google or Bing SERP). +- Listing the pages of a site (map) or fetching many pages of one site or section (crawl). - E-commerce data (Amazon, Walmart, Target product/search pages). - Social media data (Reddit, TikTok, YouTube). - A screenshot of a rendered page. @@ -77,7 +79,12 @@ the API first) or just run the real command and handle exit 3. ### 2. Authenticate The user needs a Web Data API **key** (or, on older plans, a basic auth token) from - (free account = up to 2K requests, no card). + (free account = up to 10K requests, no card). + +A basic auth token works for `scrape`, `screenshot` and target commands, but `search`, `map` and +`crawl` need an API key and reject a basic token with exit code `3`. `decodo whoami` prints +`api key:` or `token:` so you can tell which one is set. If the user only has a basic token and +asks for search, map or crawl, tell them they need an API key from the dashboard. Prefer the environment variable — it needs no interaction and is easy to scope to a session: @@ -114,21 +121,25 @@ npm install -g @decodo/cli # any platform Three surfaces, one credential. This plugin ships two of them — the MCP server (`mcp.json`) and this CLI skill — so **both may be live at once**. When they are, decide by the task, in this order: -1. **Decodo MCP tools are in your tool list** (`scrape_as_markdown`, `google_search`, …) **and - the task is a plain scrape or search** → call the MCP tool. Fewer steps, no install, no - token plumbing. Enabled toolsets default to `web,search`; the user can widen them with the - `DECODO_TOOLSETS` plugin variable. -2. **Otherwise → the `decodo` CLI.** It covers the full target surface (all ~30 targets, +1. **Decodo MCP tools are in your tool list** (`scrape_as_markdown`, `search`, `map`, …) **and + the task is a plain scrape, a search, or a map** → call the MCP tool. Fewer steps, no + install, no token plumbing. Enabled toolsets default to `web,search`; the user can widen them + with the `DECODO_TOOLSETS` plugin variable. +2. **A crawl, and a shell is available → the `decodo` CLI**, even when the MCP crawl tools are + listed. `decodo crawl ... -o pages.ndjson` keeps the pages on disk so you read only the slices + you need; MCP `crawl_results` returns up to ~50 KB per call straight into your context. With + no shell, use the MCP crawl tools (see **Search, map and crawl over MCP**). +3. **Otherwise → the `decodo` CLI.** It covers the full target surface (all ~40 targets, generated from the API schema at runtime) regardless of which MCP toolsets are enabled. Required for anything MCP cannot do: a target outside the enabled toolsets (Amazon, Walmart, Target, Reddit, TikTok, YouTube when only `web,search` is on), writing binary output to a file (`screenshot -o`), batching many URLs, or piping into `jq`/a script. The rest of this skill assumes this lane. -3. **No shell and no MCP** (Claude Desktop, claude.ai, an MCP-only client not yet configured) → +4. **No shell and no MCP** (Claude Desktop, claude.ai, an MCP-only client not yet configured) → set up MCP from [`references/mcp-setup.md`](references/mcp-setup.md), or call the **raw HTTP API** with `curl` — recipes in [`references/api-curl.md`](references/api-curl.md). That means - `curl` against Decodo's endpoint (`https://scraper-api.decodo.com/v2/scrape`) with your - token. It is never permission to `curl` the target site directly; all three surfaces are + `curl` against Decodo's API (`https://data.decodo.com/v1/...` with an API key, or + `https://scraper-api.decodo.com/v2/scrape` with a basic auth token). It is never permission to `curl` the target site directly; all three surfaces are Decodo. Do not run the same request through both lanes to compare them — it bills twice. @@ -149,12 +160,79 @@ never fall back to fetching the target site directly. ```bash decodo scrape https://example.com # markdown by default -decodo scrape https://example.com --country us # geo-target the request -decodo search "best web scraping api" --limit 3 # Google SERP, 3 result pages -decodo search "query" --engine bing --geo de # Bing, geo Germany -decodo screenshot https://example.com -o shot.png # PNG must go to a file +decodo scrape https://example.com --country US # geo-target the request +decodo search "best web scraping api" | jq -r '.organic[].link' # web search, JSON +decodo search "query" --geo DE --locale de-DE --page 2 # geo, locale, next page +decodo map https://example.com --limit 100 # site URLs, one per line +decodo screenshot https://example.com -o shot.png # PNG must go to a file +``` + +`search` always returns parsed JSON (`organic`, `top_stories`, `people_also_ask`, …) and takes +only `--geo`, `--locale`, `--page`, `--pretty` and `-o`. It has no `--engine`, `--limit` or +`--parse` (CLI 2.0+ rejects them with exit `2`). For a specific engine or its full SERP, use the +`google-search` or `bing-search` target (below), e.g. `decodo bing-search "query" --geo DE --parse`. + +### Search, map and crawl a site + +`search`, `map` and `crawl` run on the Data API and need an **API key** (see Setup step 2). + +**Map first.** `map` lists a site's URLs without fetching them: cheap, fast, small output. +Use it to see what exists, then either `scrape` the few pages you need or crawl a scoped +section. Don't crawl a whole site to find one page. + +```bash +decodo map https://docs.example.com --limit 200 -o urls.txt +wc -l urls.txt; grep -i pricing urls.txt | head +decodo map https://example.com --select-paths "/blog/*" --exclude-paths "/blog/tag/*" --limit 100 +``` + +**Scope every crawl.** Always pass `--limit`, and narrow with `--select-paths` / +`--exclude-paths` (repeatable) or `--max-depth`. Unscoped crawls are slow and burn requests. + +| Flag (map and crawl) | Effect | +| --- | --- | +| `--limit ` | Max pages. Always set it | +| `--select-paths ` | Only paths matching, e.g. `"/docs/*"`. Repeatable | +| `--exclude-paths ` | Skip matching paths. Repeatable | +| `--sitemap include\|only\|skip` | Use the sitemap with page links, only it, or ignore it | +| `--domain-filter domain\|subdomain\|all` | Which hosts to follow | +| `--geo`, `--locale` | Geo-target the requests | +| `--max-depth ` | Crawl only: link depth from the start URL | +| `--max-duration ` | Crawl only: stop the crawl after this many seconds | +| `--content-format markdown\|html` | Crawl only: page content format | +| `--js-rendering` | Map only: render JS while discovering links | + +**Start crawls async, collect later.** `decodo crawl ` returns at once and prints only the +crawl id. Keep working, then collect the pages with `crawl results --wait`, which polls until +the crawl finishes: + +```bash +id=$(decodo crawl https://docs.example.com --select-paths "/docs/*" --limit 50) +# ... other work ... +decodo crawl status "$id" # discovered/completed/failed, status +decodo crawl results "$id" --wait --progress -o pages.ndjson +decodo crawl cancel "$id" # stop early; crawled pages stay readable +``` + +For a small crawl you'll wait on anyway, start and collect in one call: +`decodo crawl --limit 20 --wait -o pages.ndjson` (the id goes to stderr). +`--wait` polls every `--poll-interval` seconds (default 5) and gives up after `--timeout` +seconds (default 3600, exit `6`); the crawl keeps running until you cancel it. +`truncated`/`cancelled` crawls print a warning; a `faulted` crawl writes its pages, then exits `7`. + +**Never print crawl output into your context.** Always write it to a file with `-o`, then read +it in slices. Each NDJSON line is one page: `url`, `depth`, `status`, `content`, … + +```bash +wc -l pages.ndjson # how many pages +jq -r '[.status, .url] | @tsv' pages.ndjson | head -50 # page list, no content +jq -r 'select(.url | test("/pricing")) | .content' pages.ndjson | head -c 4000 +jq -r 'select(.content_error) | .url' pages.ndjson # pages that failed +decodo crawl results "$id" --no-content -o index.ndjson # metadata only, no content ``` +Don't `cat` the file, and don't `head -n` it either: one line can hold a whole page. + ### Discover targets — don't guess Target subcommands are generated from the live API schema, so **list and introspect them** @@ -181,7 +259,8 @@ decodo universal https://example.com # generic target with the most flag ## Output handling (important for piping) By default a command prints the first result's `content` (markdown for `scrape`, parsed -JSON for targets called with `--parse`). Useful modifiers: +JSON for targets called with `--parse`). These modifiers apply to `scrape` and target commands. +`search`, `map` and `crawl` support a subset; check `decodo --help`. | Flag | Effect | | --- | --- | @@ -205,7 +284,7 @@ only when you actually need status/headers. decodo google-search "rust web scraping" --parse | jq -r '.results.results.organic[].title' # Extract a field from a scraped JSON page (inspect with `jq keys` first if unsure) -decodo scrape https://ip.decodo.com/json | jq -r '.proxy.ip' +decodo universal https://ip.decodo.com/json --no-markdown | jq -r '.proxy.ip' # Stream results as NDJSON; each record nests the parsed payload under `.content` decodo google-search "rust" --parse --format ndjson --full | jq -c '.content.results.results.organic' @@ -220,11 +299,31 @@ stdout is data; logs and errors go to stderr — pipes stay clean. | 0 | Success | — | | 1 | Generic error | Read stderr | | 2 | Usage error (bad flags) | Re-check `--help` | -| 3 | Auth error | Key or token missing/rejected (`No API key or auth token found.`, `Invalid credentials`, `Incorrect username or password`) → redo **Setup step 2** | +| 3 | Auth error | Key or token missing/rejected (`No API key or auth token found.`, `Invalid credentials`, `Incorrect username or password`), or a basic token on `search`/`map`/`crawl` → redo **Setup step 2** (with an API key for those three) | | 4 | Validation error | Fix the argument/flag the message names | | 5 | Rate limited | Back off and retry; reduce concurrency | | 6 | Timeout | Retry with backoff | -| 7 | Network/server error | Retry with backoff | +| 7 | Network/server error, or a `faulted` crawl | Retry with backoff | + +## Search, map and crawl over MCP + +The plugin's MCP server has the same Data API tools in its default toolsets (`search` in +`search`; `map` and the crawl tools in `web`). They also need an API key, not a basic token. + +- `search` — web search; `limit` caps organic results (10 by default). +- `map` — site URLs as a JSON list; `limit` defaults to 10, so set it and narrow with + `selectPaths` / `excludePaths`. Then `scrape_as_markdown` only the pages you need. +- Crawl is async, in four tools. Prefer the CLI when you have a shell (see **Choosing a + surface**); use these when you don't: + 1. `crawl_start` with `url`, a `limit` and path filters → returns `crawl_id` at once. + 2. `crawl_status` with `crawlId` → poll every 10–30 s, doing other work in between, + until `status` is no longer `pending`. + 3. `crawl_results` with `crawlId` → about 50 KB per call. Pass `nextCursor` back as `cursor` + until it is missing. A long page is split across calls; `contentPart` gives the range. + 4. `crawl_cancel` with `crawlId` → stop early; pages crawled so far stay readable. + +Read one `crawl_results` page at a time and keep only what the task needs, rather than +collecting every page before you start. ## Raw API fallback (no CLI, no MCP) diff --git a/plugins/decodo/skills/decodo/references/api-curl.md b/plugins/decodo/skills/decodo/references/api-curl.md index 330c6a9..6151cf6 100644 --- a/plugins/decodo/skills/decodo/references/api-curl.md +++ b/plugins/decodo/skills/decodo/references/api-curl.md @@ -25,7 +25,7 @@ and . | `query` | string | For search targets (`google_search`, `bing_search`, …) | | `markdown` | bool | Return page content as Markdown | | `parse` | bool | Return structured JSON (parse-enabled targets only) | -| `geo` | string | Country/geo code, e.g. `us`, `de` | +| `geo` | string | Country/geo code, e.g. `US`, `DE` | | `headless` | string | `png` for a screenshot, `html` for rendered HTML | ## Recipes @@ -47,7 +47,7 @@ curl -s https://data.decodo.com/v1/scrape \ -H "Authorization: Bearer $DECODO_AUTH_TOKEN" \ -H "Content-Type: application/json" \ -H "x-integration: cursor-plugin" \ - -d '{"target":"google_search","query":"rust web scraping","parse":true,"geo":"us"}' + -d '{"target":"google_search","query":"rust web scraping","parse":true,"geo":"US"}' ``` ### Screenshot (PNG, base64 in the response) @@ -60,9 +60,33 @@ curl -s https://data.decodo.com/v1/scrape \ -d '{"target":"universal","url":"https://example.com","headless":"png"}' ``` +## Search, map and crawl (Data API) + +These endpoints sit on the same host and need an API key; a basic auth token is rejected. + +```bash +H=(-H "Authorization: Bearer $DECODO_AUTH_TOKEN" -H "Content-Type: application/json" -H "x-integration: cursor-plugin") + +# Web search → parsed JSON with .organic[] +curl -s https://data.decodo.com/v1/search "${H[@]}" -d '{"query":"rust web scraping","geo":"DE"}' + +# Map → .results[].content.links[].url +curl -s https://data.decodo.com/v1/map "${H[@]}" \ + -d '{"url":"https://example.com","limit":100,"select_paths":["/blog/*"]}' + +# Crawl: start (returns crawl_id), poll status, then page through results with `next` +curl -s https://data.decodo.com/v1/crawl "${H[@]}" -d '{"url":"https://example.com","limit":50}' +curl -s https://data.decodo.com/v1/crawl//status "${H[@]}" +curl -s "https://data.decodo.com/v1/crawl/?type=content" "${H[@]}" -o pages.json # omit ?type=content for metadata only +curl -s -X POST https://data.decodo.com/v1/crawl//cancel "${H[@]}" +``` + +Write crawl pages to a file and read them with `jq`, never straight into context. + ## Basic auth token (older plans) -Same bodies and response shape; only the host, path and header change: +Same bodies and response shape for `/scrape`; only the host, path and header change. There is +no search, map or crawl on this host — those need an API key. ```bash curl -s https://scraper-api.decodo.com/v2/scrape \ diff --git a/plugins/decodo/skills/decodo/references/mcp-setup.md b/plugins/decodo/skills/decodo/references/mcp-setup.md index 091a736..0f29ea4 100644 --- a/plugins/decodo/skills/decodo/references/mcp-setup.md +++ b/plugins/decodo/skills/decodo/references/mcp-setup.md @@ -10,7 +10,8 @@ native scraping over the CLI. Decodo's MCP server is hosted at `https://mcp.deco authenticates with the same credential used everywhere else: a Web Data API **key** (`Authorization: Bearer `) or, on older plans, a **basic auth token** (`Authorization: Basic `). Get one at -(free tier: ~2K requests, no card). +(free tier: up to 10K requests, no card). `search`, `map` and the crawl tools need an API key; +a basic auth token is rejected on those. ## Hosted server — generic config @@ -29,9 +30,11 @@ Most MCP clients accept a remote server by URL + headers. Add: ### Toolsets (optional, recommended) -The server exposes ~30 tools across five toolsets: `web`, `search`, `ecommerce`, -`social_media`, `ai`. Enabling only what you need keeps tool selection clean and the agent -faster. On the hosted URL, scope with a query param: +Tools are grouped into five toolsets: `web` (scraping, screenshots, map and crawl), `search` +(web and SERP search), `ecommerce`, `social_media` and `ai`. The hosted server hides some +tools, so the tool list your client shows is the source of truth; don't assume a tool exists +because the npm package or its README lists it. Enabling only what you need keeps tool +selection clean and the agent faster. On the hosted URL, scope with a query param: ``` https://mcp.decodo.com/mcp?toolsets=web,search