[Bug]: Docker image ships without pypdf, so PDFContentScrapingStrategy fails at runtime

[bong-u]

crawl4ai version

0.9.2

Expected Behavior

After #1815 added PDFContentScrapingStrategy to ALLOWED_DESERIALIZE_TYPES,
POST /crawl with a PDF URL and that strategy should return extracted PDF content from the official Docker image.

Current Behavior

Deserialization works now, but the request still fails. Two blockers remain that #1815 did not cover.

1. The Docker image has no pypdf.

pypdf is only a [pdf] extra (pyproject.toml:62) and is absent from deploy/docker/requirements.txt. With the default INSTALL_TYPE=default the Dockerfile runs pip install "/tmp/project" (Dockerfile:168), no extras, so the strategy raises at runtime:

ImportError: pypdf is required for PDF processing. Install with 'pip install crawl4ai[pdf]'

(crawl4ai/processors/pdf/processor.py:64,80,131,239,422)

INSTALL_TYPE=all pulls it in, but also torch, transformers and nltk, just to get one pure-Python dependency.

2. /crawl/stream silently discards the strategy.

deploy/docker/api.py:876 overwrites whatever the client sent:

crawler_config.scraping_strategy = LXMLWebScrapingStrategy()

The non-streaming handle_crawl_request path has no such line, so the same payload behaves differently on the two endpoints and the streaming one reports no error.

Related, for PDFs served as attachments: async_crawler_strategy.py:770-778 only tolerates net::ERR_ABORTED when browser_config.accept_downloads is set. PDFs that Chromium renders inline, like arxiv's, navigate fine. Ones sent with Content-Disposition: attachment raise RuntimeError: Failed on navigating ACS-GOTO before scraping runs. See also #1367.

Is this reproducible?

Yes

Inputs Causing the Bug

`POST /crawl` against https://arxiv.org/pdf/2310.08535 with `crawler_config.params.scraping_strategy.type = "PDFContentScrapingStrategy"`.

Steps to Reproduce

1. Run the official crawl4ai Docker image.
2. `POST /crawl` with the payload below, get `ImportError: pypdf is required`.
3. Send the same payload to `/crawl/stream`, get HTML-scraped empty content instead, with no error.

Code snippets

{
  "urls": ["https://arxiv.org/pdf/2310.08535"],
  "crawler_config": {
    "type": "CrawlerRunConfig",
    "params": {
      "cache_mode": "bypass",
      "stream": false,
      "scraping_strategy": {
        "type": "PDFContentScrapingStrategy",
        "params": {"extract_images": false, "batch_size": 8}
      }
    }
  }
}

OS

Linux (Docker)

Python version

3.12

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

ImportError: pypdf is required for PDF processing. Install with 'pip install crawl4ai[pdf]

[nightcityblade]

Hi, I'd like to work on this. I'll submit a PR shortly.