[Bug]: lxml~=5.3 is too old, can't install crawl4ai & scrapling at the same time

[olokshyn]

crawl4ai version

0.8.9

Expected Behavior

Update the lxml dependency to allow versions >=6.0. This will allow crawl4ai to be used with other lxml-dependent packages like scrapling.

Current Behavior

crawl4ai depends on an old version of the lxml >=5.3,<6.0.
This means crawl4ai cannot be installed along any other dependencies that use lxml>=6.0

One example of that is Scrapling, which is excellent at bypassing Cloudflare bot checks.

Installation with poetry fails due to incompatible lxml dependencies:

$ poetry init
...
$ poetry add scrapling
...
$ poetry add crawl4ai
Using version ^0.8.9 for crawl4ai

Updating dependencies
Resolving dependencies... (2.2s)

Because no versions of crawl4ai match >0.8.9,<0.9.0
 and crawl4ai (0.8.9) depends on lxml (>=5.3,<6.0), crawl4ai (>=0.8.9,<0.9.0) requires lxml (>=5.3,<6.0).
And because scrapling (0.4.9) depends on lxml (>=6.1.1)
 and no versions of scrapling match >0.4.9,<0.5.0, crawl4ai (>=0.8.9,<0.9.0) is incompatible with scrapling (>=0.4.9,<0.5.0).
So, because test depends on both scrapling (>=0.4.9,<0.5.0) and crawl4ai (^0.8.9), version solving failed.

Installation with pip just replaces lxml with version 5.4.0, which means breaking lxml API for all other packages in the environment.

As a side note, this old lxml dependency causes another issue on Windows: #1903

Is this reproducible?

Yes

Inputs Causing the Bug

Steps to Reproduce

See the Current behavior

Code snippets

OS

Any. It's a dependency configuration issue.

Python version

Python 3.13.13. All versions are affected.

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

No response

[nightcityblade]

Hi, I'd like to work on this. I'll submit a PR shortly.