Fix SDK to working state (port gaps vs sdk-ts): validation, batch, target_meta, packaging, CI

.github/workflows/test.yml

@@ -0,0 +1,26 @@
+name: Test
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: pip install -e ".[dev]"
+
+      - name: Generate types
+        run: python -m decodo.codegen.codegen
+
+      - name: Pytest
+        run: pytest

.github/workflows/worklfow.yml

@@ -12,13 +12,20 @@ jobs:
     environment: pypi
     permissions:
       id-token: write
+      contents: read
     steps:
       - uses: actions/checkout@v4
 
       - uses: actions/setup-python@v5
         with:
           python-version: "3.12"
 
+      - name: Generate types
+        run: |
+          pip install httpx pydantic jsonschema
+          pip install -e .
+          python -m decodo.codegen.codegen
+
       - name: Build
         run: |
           pip install build

.gitignore

@@ -33,5 +33,8 @@ htmlcov/
 # Inputs (fetched at codegen time)
 inputs/decodo.ir.json
 
+# Generated types (run `python -m decodo.codegen.codegen` to regenerate)
+src/decodo/generated/
+
 # OS
 .DS_Store

README.md

@@ -55,18 +55,20 @@ pip install decodo-sdk
 touch main.py
 ```
 
-Get a Web Scraping API basic authentication token from the [Decodo dashboard](https://dashboard.decodo.com/welcome) and use it in the following example:
+Get your Web Scraping API token from the [Decodo dashboard](https://dashboard.decodo.com/welcome). The token is the base64-encoded `user:password` value from the Basic Auth credentials shown in the dashboard.
 
 ```python
 # main.py
-from decodo import DecodoClient, Target
+from decodo import DecodoClient, DecodoConfig, WebScrapingApiConfig
 
 client = DecodoClient(
-    web_scraping_api={"token": "<basic_auth_token>"}
+    DecodoConfig(
+        web_scraping_api=WebScrapingApiConfig(token="<basic_auth_token>"),
+    )
 )
 
 result = client.web_scraping_api.scrape({
-    "target": Target.GoogleSearch,
+    "target": "google_search",
     "query": "coffee shops",
     "geo": "United States",
     "parse": True,
@@ -80,6 +82,46 @@ Run the script:
 python main.py
 ```
 
+### With typed parameters (recommended)
+
+Typed parameter classes are included in the package — no extra steps needed after `pip install decodo-sdk`:
+
+```python
+from decodo import DecodoClient, DecodoConfig, GoogleSearchParams, Target, WebScrapingApiConfig
+
+client = DecodoClient(
+    DecodoConfig(
+        web_scraping_api=WebScrapingApiConfig(token="<basic_auth_token>"),
+    )
+)
+
+result = client.web_scraping_api.scrape(
+    GoogleSearchParams(
+        target=Target.GoogleSearch,
+        query="coffee shops",
+        geo="United States",
+        parse=True,
+    )
+)
+print(result)
+```
+
+### Updating types to a newer schema
+
+The types bundled in the package reflect the schema at release time. To update them to the latest schema without waiting for a new release, run the type generator:
+
+```bash
+python -m decodo.codegen.codegen --out-dir ./decodo_generated
+```
+
+Then import from that directory instead:
+
+```python
+from decodo_generated.targets import GoogleSearchParams
+```
+
+> The directory name passed to `--out-dir` becomes the import namespace. `./decodo_generated` → `from decodo_generated.targets import ...`. You can use any name, but keep it consistent across your project.
+
 <details>
 <summary>Example response</summary>
 
@@ -162,36 +204,34 @@ python main.py
 ## Configuration
 
 ```python
-from decodo import DecodoClient
+from decodo import DecodoClient, DecodoConfig, WebScrapingApiConfig
 
 client = DecodoClient(
-    web_scraping_api={"token": "<basic_auth_token>"},
-    timeout_ms=120_000,  # optional, request timeout in ms (default: 180000)
+    DecodoConfig(
+        web_scraping_api=WebScrapingApiConfig(token="<basic_auth_token>"),
+        timeout_ms=120_000,  # optional, request timeout in ms (default: 180000)
+    )
 )
 ```
 
 | Parameter | Description |
 | --- | --- |
-| `token` | Web Scraping API basic authentication token |
+| `token` | Web Scraping API basic auth token - the base64-encoded `user:password` string from the Decodo dashboard |
 | `timeout_ms` | Request timeout in milliseconds (default: 180000) |
 
 ## Web Scraping API
 
 Access the API via `client.web_scraping_api`.
 
-The snippets below assume you have already imported `Target` (and `DecodoClient` where a client is constructed), for example:
-
-```python
-from decodo import DecodoClient, Target
-```
+The snippets below assume you have already constructed a client. See [Configuration](#configuration) for how to build one.
 
 ### Sync scrape
 
 Waits for the scraping result before returning:
 
 ```python
 result = client.web_scraping_api.scrape({
-    "target": Target.AmazonProduct,
+    "target": "amazon_product",
     "query": "B09H74FXNW",
     "parse": True,
 })
@@ -203,7 +243,7 @@ Creates a scraping task and returns immediately. Poll separately for task status
 
 ```python
 task = client.web_scraping_api.scrape_async({
-    "target": Target.GoogleSearch,
+    "target": "google_search",
     "query": "laptop reviews",
     "parse": True,
 })
@@ -220,7 +260,7 @@ Send multiple queries or URLs in a single request:
 
 ```python
 batch = client.web_scraping_api.scrape_batch({
-    "target": Target.GoogleSearch,
+    "target": "google_search",
     "query": ["coffee", "tea", "juice"],
     "parse": True,
 })
@@ -241,9 +281,14 @@ Each target accepts one primary input parameter (`url`, `query`, `product_id`, o
 | `Target.GoogleSearch` | Google Search results for a query | `{"target": Target.GoogleSearch, "query": "coffee shops"}` |
 | `Target.GoogleMaps` | Google Maps search results | `{"target": Target.GoogleMaps, "query": "coffee shops brooklyn"}` |
 | `Target.GoogleShoppingSearch` | Google Shopping search results | `{"target": Target.GoogleShoppingSearch, "query": "laptop"}` |
+| `Target.GoogleShoppingProduct` | Google Shopping product page | `{"target": Target.GoogleShoppingProduct, "query": "B09H74FXNW"}` |
 | `Target.GoogleSuggest` | Google Autocomplete suggestions | `{"target": Target.GoogleSuggest, "query": "coffee"}` |
 | `Target.GoogleLens` | Google Lens reverse image search | `{"target": Target.GoogleLens, "query": "https://example.com/cat.jpg"}` |
+| `Target.GoogleTravelHotels` | Google Travel hotel listings | `{"target": Target.GoogleTravelHotels, "query": "hotels in paris"}` |
+| `Target.GoogleTrendsExplore` | Google Trends explore data | `{"target": Target.GoogleTrendsExplore, "query": "coffee"}` |
+| `Target.GoogleAds` | Google Ads results for a query | `{"target": Target.GoogleAds, "query": "laptop"}` |
 | `Target.BingSearch` | Bing Search results | `{"target": Target.BingSearch, "query": "electric vehicles"}` |
+| `Target.Bing` | Raw Bing URL scraping | `{"target": Target.Bing, "url": "https://bing.com/search?q=laptop"}` |
 
 ### eCommerce
 
@@ -252,8 +297,15 @@ Each target accepts one primary input parameter (`url`, `query`, `product_id`, o
 | `Target.AmazonProduct` | Amazon product detail page by ASIN | `{"target": Target.AmazonProduct, "query": "B09H74FXNW"}` |
 | `Target.AmazonSearch` | Amazon search results | `{"target": Target.AmazonSearch, "query": "laptop"}` |
 | `Target.AmazonPricing` | Amazon pricing and offers | `{"target": Target.AmazonPricing, "query": "B09H74FXNW"}` |
+| `Target.AmazonSellers` | Amazon seller listings | `{"target": Target.AmazonSellers, "query": "B09H74FXNW"}` |
+| `Target.AmazonBestsellers` | Amazon bestsellers by category | `{"target": Target.AmazonBestsellers, "query": "electronics"}` |
 | `Target.WalmartProduct` | Walmart product page by product ID | `{"target": Target.WalmartProduct, "product_id": "15296401808"}` |
+| `Target.WalmartSearch` | Walmart search results | `{"target": Target.WalmartSearch, "query": "laptop"}` |
+| `Target.Walmart` | Raw Walmart URL scraping | `{"target": Target.Walmart, "url": "https://walmart.com/ip/15296401808"}` |
 | `Target.TargetProduct` | Target.com product page by product ID | `{"target": Target.TargetProduct, "product_id": "92186007"}` |
+| `Target.TargetSearch` | Target.com search results | `{"target": Target.TargetSearch, "query": "laptop"}` |
+| `Target.Target` | Raw Target.com URL scraping | `{"target": Target.Target, "url": "https://target.com/p/-/A-92186007"}` |
+| `Target.LowesSearch` | Lowe's search results | `{"target": Target.LowesSearch, "query": "drill"}` |
 | `Target.Ecommerce` | Generic eCommerce page with parser | `{"target": Target.Ecommerce, "url": "https://example.com/product/123"}` |
 
 ### Social media
@@ -262,9 +314,19 @@ Each target accepts one primary input parameter (`url`, `query`, `product_id`, o
 | --- | --- | --- |
 | `Target.RedditPost` | Reddit post by URL | `{"target": Target.RedditPost, "url": "https://reddit.com/r/nba/..."}` |
 | `Target.RedditSubreddit` | Reddit subreddit by URL | `{"target": Target.RedditSubreddit, "url": "https://reddit.com/r/nba/"}` |
+| `Target.RedditUser` | Reddit user profile by URL | `{"target": Target.RedditUser, "url": "https://reddit.com/user/example/"}` |
 | `Target.YoutubeVideo` | YouTube video by ID | `{"target": Target.YoutubeVideo, "query": "dFu9aKJoqGg"}` |
 | `Target.YoutubeSearch` | YouTube search results | `{"target": Target.YoutubeSearch, "query": "ambient music"}` |
+| `Target.YoutubeSearchMax` | YouTube search results (extended) | `{"target": Target.YoutubeSearchMax, "query": "ambient music"}` |
+| `Target.YoutubeMetadata` | YouTube video metadata by ID | `{"target": Target.YoutubeMetadata, "query": "dFu9aKJoqGg"}` |
+| `Target.YoutubeTranscript` | YouTube video transcript by ID | `{"target": Target.YoutubeTranscript, "query": "dFu9aKJoqGg"}` |
+| `Target.YoutubeSubtitles` | YouTube video subtitles by ID | `{"target": Target.YoutubeSubtitles, "query": "dFu9aKJoqGg"}` |
+| `Target.YoutubeChannel` | YouTube channel by URL | `{"target": Target.YoutubeChannel, "url": "https://youtube.com/@mkbhd"}` |
 | `Target.TiktokPost` | TikTok post by URL | `{"target": Target.TiktokPost, "url": "https://www.tiktok.com/@nba/video/..."}` |
+| `Target.TiktokShopSearch` | TikTok Shop search results | `{"target": Target.TiktokShopSearch, "query": "wireless earbuds"}` |
+| `Target.TiktokShopProduct` | TikTok Shop product page | `{"target": Target.TiktokShopProduct, "url": "https://www.tiktok.com/view/product/..."}` |
+| `Target.Tiktok` | Raw TikTok URL scraping | `{"target": Target.Tiktok, "url": "https://www.tiktok.com/@nba"}` |
+| `Target.InstagramGraphqlProfile` | Instagram profile via GraphQL | `{"target": Target.InstagramGraphqlProfile, "query": "nba"}` |
 
 ### AI tools
 
@@ -275,6 +337,16 @@ Each target accepts one primary input parameter (`url`, `query`, `product_id`, o
 | `Target.Gemini` | Gemini response for a prompt | `{"target": Target.Gemini, "prompt": "What are the top three dog breeds?"}` |
 | `Target.GoogleAiMode` | Google AI Mode response | `{"target": Target.GoogleAiMode, "query": "What are the top three dog breeds?"}` |
 
+### Other
+
+| Target | Description | Example |
+| --- | --- | --- |
+| `Target.Bbb` | Better Business Bureau listing by URL | `{"target": Target.Bbb, "url": "https://bbb.org/us/ny/new-york/..."}` |
+| `Target.Autotrader` | Autotrader listing by URL | `{"target": Target.Autotrader, "url": "https://autotrader.com/cars-for-sale/..."}` |
+| `Target.Mobile` | Mobile.de listing by URL | `{"target": Target.Mobile, "url": "https://mobile.de/auto/..."}` |
+| `Target.Airbnb` | Airbnb listing by URL | `{"target": Target.Airbnb, "url": "https://airbnb.com/rooms/12345"}` |
+| `Target.AppleAppStore` | Apple App Store app by URL | `{"target": Target.AppleAppStore, "url": "https://apps.apple.com/app/id12345"}` |
+
 ### Universal scraping
 
 | Target | Description | Example |
@@ -296,27 +368,24 @@ The SDK raises typed errors that map to API error codes:
 
 ```python
 from decodo import (
-    DecodoClient,
-    DecodoError,
     AuthenticationError,
     RateLimitError,
     ValidationError,
     TimeoutError,
-    Target,
 )
 
 try:
     client.web_scraping_api.scrape({
-        "target": Target.GoogleSearch,
+        "target": "google_search",
         "query": "test",
         "parse": True,
     })
 except AuthenticationError:
-    pass  # 401/403 — bad credentials
+    pass  # 401/403 - bad credentials
 except RateLimitError:
-    pass  # 429 — too many requests
+    pass  # 429 - too many requests
 except ValidationError as err:
-    print(err.errors)  # 422 — invalid parameters
+    print(err.errors)  # 422 - invalid parameters
 except TimeoutError:
     pass  # request timed out
 ```
@@ -338,4 +407,3 @@ Build scraping workflows with the Decodo Web Scraping API:
 ## License
 
 Released under the [MIT License](https://github.com/Decodo/Decodo/blob/master/LICENSE).
->>>>>>> 03b68da (Initial sdk setup)

examples/web_scraping_api/README.md

@@ -0,0 +1,25 @@
+# Examples
+
+These examples use typed parameter classes (`GoogleSearchParams`, `AmazonProductParams`, etc.) which require running the type generator first:
+
+```bash
+python -m decodo.codegen.codegen
+```
+
+To run any example without the generator, replace the typed params with a plain dict:
+
+```python
+# instead of:
+result = client.web_scraping_api.scrape(
+    GoogleSearchParams(target=Target.GoogleSearch, query="coffee shops", parse=True)
+)
+
+# use:
+result = client.web_scraping_api.scrape({
+    "target": "google_search",
+    "query": "coffee shops",
+    "parse": True,
+})
+```
+
+See the [root README](../../README.md) for the full list of target strings.

examples/web_scraping_api/batch/google_search_batch.py

@@ -5,7 +5,7 @@
 from decodo import (
     DecodoClient,
     DecodoConfig,
-    GoogleSearchParams,
+    GoogleSearchBatchParams,
     Target,
     WebScrapingApiConfig,
 )
@@ -19,25 +19,23 @@
 )
 
 metadata = client.web_scraping_api.scrape_batch(
-    GoogleSearchParams(
+    GoogleSearchBatchParams(
         target=Target.GoogleSearch,
-        query=['shoes', 'laptop'],
+        query=["shoes", "laptop"],
         parse=True,
     )
 )
 
-print('Polling for results...')
-
 while True:
-    print('Polling for results...')
-    queries = metadata.get('queries') or []
+    print("Polling for results...")
+    queries = metadata.get("queries") or []
     if not queries:
         break
-    first_task_id = queries[0].get('id')
+    first_task_id = queries[0].get("id")
     if not first_task_id:
         break
     results = client.web_scraping_api.get_results(first_task_id)
     if results:
-        print(json.dumps(results['results'][0]['content'], indent=2))
+        print(json.dumps(results["results"][0]["content"], indent=2))
         break
     time.sleep(3)

inputs/README.md

@@ -3,4 +3,4 @@
 This directory contains files fetched at codegen time and is excluded from version control.
 
 - `decodo.ir.json` — the Intermediate Representation (IR) fetched from the Decodo GCS bucket.
-  Run `decodo-codegen` (or `python -m decodo.codegen.codegen`) to populate it.
+  Run `python -m decodo.codegen.codegen` to populate it.