Skip to content

fix(docker): support PDF scraping by default - #2130

Open
nightcityblade wants to merge 1 commit into
unclecode:developfrom
nightcityblade:fix/issue-2127
Open

fix(docker): support PDF scraping by default#2130
nightcityblade wants to merge 1 commit into
unclecode:developfrom
nightcityblade:fix/issue-2127

Conversation

@nightcityblade

Copy link
Copy Markdown
Contributor

Summary

Fixes #2127.

The default Docker image now installs the PDF processor dependency, and the streaming crawl handler preserves the scraping strategy supplied by the client instead of replacing it with LXMLWebScrapingStrategy.

List of files changed and why

  • deploy/docker/requirements.txt - Install pypdf in the default Docker image.
  • deploy/docker/api.py - Preserve the deserialized scraping strategy for streaming crawls.
  • tests/test_issue_2127_docker_pdf.py - Add regressions for the Docker dependency and streaming-handler behavior.

How Has This Been Tested?

  • .venv/bin/pytest -q tests/test_issue_2127_docker_pdf.py tests/test_pr_1795_1798_1734.py::TestDeepCrawlStreamBranching (7 passed)
  • .venv/bin/black --check --target-version py312 --fast --line-ranges 867-880 deploy/docker/api.py
  • .venv/bin/black --check --target-version py312 --fast tests/test_issue_2127_docker_pdf.py
  • .venv/bin/python -m py_compile deploy/docker/api.py tests/test_issue_2127_docker_pdf.py
  • uv pip install --dry-run -r deploy/docker/requirements.txt

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas — N/A; the change is straightforward and the regression tests are named descriptively.
  • I have made corresponding changes to the documentation — N/A; this restores the documented Docker API behavior.
  • I have added/updated unit tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant