diff --git a/README.md b/README.md index 41f0e05e7f3..01e7847d8e3 100755 --- a/README.md +++ b/README.md @@ -8,9 +8,7 @@ -

SeleniumBase

- -

SeleniumBase

+

SeleniumBase

All-in-one Browser Automation Framework:
Web Crawling / Testing / Scraping / Stealth

@@ -52,21 +50,17 @@

-πŸ“Š SeleniumBase is a complete framework for browser automation and testing with Python and pytest. Includes stealth options and other advanced features. - -πŸ™ Stealth modes: UC Mode and CDP Mode can bypass bot-detection, handle CAPTCHAs, and call methods from the Chrome Devtools Protocol. CDP Mode includes Stealthy Playwright Mode, which makes Playwright stealthy. - -πŸ“š The [SeleniumBase/examples/](https://github.com/seleniumbase/SeleniumBase/tree/master/examples) folder includes over 100 ready-to-run examples. Examples that start with `test_` or end with `_test.py`/`_tests.py` are specifically designed to run with `pytest`. Other examples run directly with raw `python` (those files generally start with `raw_` to avoid confusion). +-------- -πŸ₯· Stealthy examples are located in [SeleniumBase/examples/cdp_mode/](https://github.com/seleniumbase/SeleniumBase/tree/master/examples/cdp_mode). +πŸ₯· Stealth modes: UC Mode / CDP Mode can bypass bot-detection, handle CAPTCHAs, and call methods from the Chrome Devtools Protocol. CDP Mode includes Stealthy Playwright Mode, which makes Playwright stealthy. --------- +πŸ§ͺ End-to-End Testing examples are located in [./examples/](https://github.com/seleniumbase/SeleniumBase/blob/master/examples). -

βš™οΈ Stealthy architecture flowchart:

+πŸ™ CDP Mode (stealth) examples are located in [./examples/cdp_mode/](https://github.com/seleniumbase/SeleniumBase/tree/master/examples/cdp_mode). -Stealthy architecture flowchart +🎭 Stealthy Playwright examples are located in [./examples/cdp_mode/playwright/](https://github.com/seleniumbase/SeleniumBase/blob/master/examples/cdp_mode/playwright/). -(For maximum stealth, use CDP Mode, which includes Stealthy Playwright Mode) +πŸ“š Examples that have `test_` or `_test` in the filename are specifically designed to run with `pytest` / `pynose`. Other examples run directly with raw `python` (those files generally start with `raw_` to avoid confusion). -------- @@ -76,18 +70,22 @@ from seleniumbase import sb_cdp sb = sb_cdp.Chrome(incognito=True) -sb.goto("www.planetminecraft.com/account/sign_in") -sb.solve_captcha() -# "LOG IN" is enabled when the CAPTCHA is bypassed -sb.wait_for_element_absent("input[disabled]") -sb.sleep(2) +sb.goto("https://www.planetminecraft.com/account") +sb.assert_element('input[name="login"]') +sb.type('input[name="email"]', "test@example.com") +sb.type('input[name="password"]', "Fake_Password") +sb.click("input#autologin") # The checkbox +if sb.is_element_visible("input[disabled]"): + sb.solve_captcha() # Enables the input +sb.assert_element_absent("input[disabled]") +sb.sleep(1.5) sb.quit() ``` > `python raw_cdp_planetmc.py`

-Cloudflare Turnstile bypassed successfully +Cloudflare Turnstile bypassed successfully

βœ… sb.solve_captcha() handles CAPTCHAs that aren't bypassed automatically.
(If called when there's no CAPTCHA on the page, then nothing happens / no-op.) @@ -99,51 +97,22 @@ sb.quit() ```python from seleniumbase import sb_cdp -sb = sb_cdp.Chrome() +sb = sb_cdp.Chrome(fast=True) sb.goto("https://news.ycombinator.com/submitted?id=seleniumbase") elements = sb.find_elements("span.titleline > a") for element in elements: - print("* %s" % element.text) + print("* " + element.text) ``` > `python raw_cdp_yc_news.py`

-SeleniumBase +SeleniumBase

----- - -

πŸ“ Here’s SeleniumBase/examples/cdp_mode/raw_cdp_methods.py:
(It performs several different actions on the web page)

- -```python -from seleniumbase import sb_cdp - -sb = sb_cdp.Chrome(fast=True) -sb.goto("https://seleniumbase.io/demo_page") -sb.type("input", "Quickly type text!") -sb.press_keys("textarea", "Slowly type text!") -sb.click("#myButton") -sb.set_value("input#mySlider", "100") -sb.click_visible_elements("input.checkBoxClassB") -sb.select_option_by_text("#mySelect", "Set to 75%") -sb.hover_and_click("#myDropdown", "#dropOption2") -sb.click("#checkBox1") -sb.drag_and_drop("img#logo", "div#drop2") -sb.nested_click("iframe#myFrame3", ".fBox") -sb.highlight("#myButton") -sb.quit() -``` - -> `python raw_cdp_methods.py` - -SeleniumBase Example - -ℹ️ Setting `fast=True` lets your automation go at full speed, which may be useful if maximum stealth isn't your goal. - ----- +-------- -

πŸ“ This example bypasses Cloudflare's challenge page with UC + CDP Mode:
(If the Turnstile isn't bypassed automatically, sb.solve_captcha() handles it.)

+

πŸ“ This example bypasses Cloudflare's challenge page with UC + CDP Mode:

```python from seleniumbase import SB @@ -160,22 +129,14 @@ with SB(uc=True, test=True, locale="en") as sb: > `python raw_gitlab.py`

-SeleniumBase +SeleniumBase

-------- -πŸ§ͺ End-to-End Testing examples are located in [./examples/](https://github.com/seleniumbase/SeleniumBase/blob/master/examples). - -πŸ™ Stealthy CDP Mode examples are located in [./examples/cdp_mode/](https://github.com/seleniumbase/SeleniumBase/tree/master/examples/cdp_mode). - -🎭 Stealthy Playwright examples are located in [./examples/cdp_mode/playwright/](https://github.com/seleniumbase/SeleniumBase/blob/master/examples/cdp_mode/playwright/). - --------- -

🌐 CLI Options for Supported Chromium Browsers

-

πŸ’‘ You can set the Chromium browser to use via command line parameters:

+

πŸ’‘ You can set which Chromium browser to use via command-line parameters:

```zsh python SCRIPT.py --chromium # Use the unbranded Chromium browser @@ -194,11 +155,13 @@ sb = sb_cdp.Chrome(use_chromium=True) -------- -

πŸ§ͺ Comprehensive E2E Testing with pytest:

+

βš™οΈ Stealthy architecture flowchart:

-πŸ“š The [SeleniumBase/examples/](https://github.com/seleniumbase/SeleniumBase/tree/master/examples) folder includes over 150 ready-to-run examples of E2E testing. Examples that start with `test_` or end with `_test.py`/`_tests.py` run with `pytest`. Other examples run directly with raw `python` (those generally start with `raw_` to avoid confusion). +Stealthy architecture flowchart +-------- +

πŸ§ͺ Comprehensive E2E Testing with pytest:

πŸ“ This example tests an e-commerce site with pytest:

@@ -228,6 +191,8 @@ class MyTestClass(BaseCase): SeleniumBase Test +πŸ₯· For stealth with `pytest`, add `--uc` as a command-line option. + --------

πŸ“ This example tests another e-commerce site with pytest:

@@ -254,7 +219,7 @@ pytest test_demo_site.py > Easy to type, click, select, toggle, drag & drop, and more. -(For more examples, see the SeleniumBase/examples/ folder.) +πŸ“š For more examples, see the SeleniumBase/examples/ folder. -------- diff --git a/examples/cdp_mode/playwright/raw_clearcote_sync.py b/examples/cdp_mode/playwright/raw_clearcote_sync.py index cf320e985f3..76c7875754d 100644 --- a/examples/cdp_mode/playwright/raw_clearcote_sync.py +++ b/examples/cdp_mode/playwright/raw_clearcote_sync.py @@ -26,3 +26,7 @@ score = sb.get_text("div.text-successText") print(f" ❌ Fingerprint tampering detected! Score: {score}") sb.sleep(1) + folder = "downloaded_files" + file_name = "c_audit_results_pw.pdf" + sb.save_as_pdf(file_name, folder) + print('"./%s/%s" was saved!' % (folder, file_name)) diff --git a/examples/cdp_mode/playwright/raw_footlocker_sync.py b/examples/cdp_mode/playwright/raw_footlocker_sync.py index f852aea986d..671adcebb70 100644 --- a/examples/cdp_mode/playwright/raw_footlocker_sync.py +++ b/examples/cdp_mode/playwright/raw_footlocker_sync.py @@ -3,29 +3,25 @@ sb = sb_cdp.Chrome(locale="en", ad_block=True) sb.goto("https://www.footlocker.com/") +sb.click_if_visible('button[id*="Agree"]', timeout=3) +sb.click('input[name="query"]') +search = "Nike Shoes" +sb.press_keys('input[name="query"]', search) endpoint_url = sb.get_endpoint_url() with sync_playwright() as p: browser = p.chromium.connect_over_cdp(endpoint_url) page = browser.contexts[0].pages[0] input_field = 'input[name="query"]' - page.wait_for_selector(input_field) - sb.sleep(1.5) - sb.click_if_visible('button[id*="Agree"]') - sb.sleep(0.6) - page.click(input_field) - sb.sleep(0.6) - search = "Nike Shoes" - page.type(input_field, search) sb.sleep(1.2) page.click('ul[id*="typeahead"] li div') sb.sleep(3.5) - elements = sb.select_all("a[data-productcard]") - if elements: + items = page.locator("a[data-productcard]") + count = items.count() + if count: print('**** Found results for "%s": ****' % search) else: print('**** No results found for "%s". ****' % search) - for element in elements: + for i in range(items.count()): print("------------------ >>>") - print("* " + element.text) - sb.sleep(2) + print("* " + items.nth(i).inner_text().replace("\n", " ")) diff --git a/examples/cdp_mode/playwright/raw_planetmc_sync.py b/examples/cdp_mode/playwright/raw_planetmc_sync.py index d18aaf48e94..577f152c71e 100644 --- a/examples/cdp_mode/playwright/raw_planetmc_sync.py +++ b/examples/cdp_mode/playwright/raw_planetmc_sync.py @@ -1,14 +1,22 @@ +from playwright.sync_api import expect from playwright.sync_api import sync_playwright from seleniumbase import sb_cdp -sb = sb_cdp.Chrome() +sb = sb_cdp.Chrome(incognito=True) endpoint_url = sb.get_endpoint_url() with sync_playwright() as p: browser = p.chromium.connect_over_cdp(endpoint_url) page = browser.contexts[0].pages[0] - page.goto("https://www.planetminecraft.com/account/sign_in/") - sb.solve_captcha() - input_disabled = page.locator("input[disabled]") - input_disabled.wait_for(state="hidden", timeout=5000) - page.wait_for_timeout(2000) + page.goto("https://www.planetminecraft.com/account") + expect(page.locator('input[name="login"]')).to_be_visible() + + email_field = page.locator('input[name="email"]') + email_field.press_sequentially("test@example.com", delay=55) + page.locator('input[name="password"]').fill("Fake_Password") + page.locator("input#autologin").click(force=True) + + if page.locator('input[disabled]').is_visible(): + sb.solve_captcha() # Enables the input button + expect(page.locator("input[disabled]")).to_be_hidden() + page.wait_for_timeout(1500) diff --git a/examples/cdp_mode/playwright/raw_yc_news_sync.py b/examples/cdp_mode/playwright/raw_yc_news_sync.py index f6364ce83b8..b73d753732b 100644 --- a/examples/cdp_mode/playwright/raw_yc_news_sync.py +++ b/examples/cdp_mode/playwright/raw_yc_news_sync.py @@ -1,7 +1,7 @@ from playwright.sync_api import sync_playwright from seleniumbase import sb_cdp -sb = sb_cdp.Chrome() +sb = sb_cdp.Chrome(fast=True) endpoint_url = sb.get_endpoint_url() with sync_playwright() as p: diff --git a/examples/cdp_mode/raw_cdp_planetmc.py b/examples/cdp_mode/raw_cdp_planetmc.py index 87f7d9875a6..e3ed5ff076d 100644 --- a/examples/cdp_mode/raw_cdp_planetmc.py +++ b/examples/cdp_mode/raw_cdp_planetmc.py @@ -1,9 +1,13 @@ from seleniumbase import sb_cdp sb = sb_cdp.Chrome(incognito=True) -sb.goto("www.planetminecraft.com/account/sign_in") -sb.solve_captcha() -# "LOG IN" is enabled when the CAPTCHA is bypassed -sb.wait_for_element_absent("input[disabled]") -sb.sleep(2) +sb.goto("https://www.planetminecraft.com/account") +sb.assert_element('input[name="login"]') +sb.type('input[name="email"]', "test@example.com") +sb.type('input[name="password"]', "Fake_Password") +sb.click("input#autologin") # The checkbox +if sb.is_element_visible("input[disabled]"): + sb.solve_captcha() # Enables the input button +sb.assert_element_absent("input[disabled]") +sb.sleep(1.5) sb.quit() diff --git a/examples/cdp_mode/raw_cdp_yc_news.py b/examples/cdp_mode/raw_cdp_yc_news.py index c6ebb09d1d8..798a268ffd9 100644 --- a/examples/cdp_mode/raw_cdp_yc_news.py +++ b/examples/cdp_mode/raw_cdp_yc_news.py @@ -1,7 +1,7 @@ from seleniumbase import sb_cdp -sb = sb_cdp.Chrome() +sb = sb_cdp.Chrome(fast=True) sb.goto("https://news.ycombinator.com/submitted?id=seleniumbase") elements = sb.find_elements("span.titleline > a") for element in elements: - print("* %s" % element.text) + print("* " + element.text) diff --git a/examples/cdp_mode/raw_footlocker.py b/examples/cdp_mode/raw_footlocker.py index 0ebca7edbf4..0c282f2cafa 100644 --- a/examples/cdp_mode/raw_footlocker.py +++ b/examples/cdp_mode/raw_footlocker.py @@ -3,14 +3,11 @@ with SB(uc=True, test=True, locale="en", ad_block=True) as sb: sb.activate_cdp_mode() sb.goto("https://www.footlocker.com/") - sb.sleep(2.5) - sb.click_if_visible('button[id*="Agree"]') - sb.sleep(0.6) + sb.click_if_visible('button[id*="Agree"]', timeout=3) sb.click('input[name="query"]') - sb.sleep(0.6) search = "Nike Shoes" sb.press_keys('input[name="query"]', search) - sb.sleep(2.5) + sb.sleep(1.2) sb.click('ul[id*="typeahead"] li div') sb.sleep(3.5) elements = sb.select_all("a[data-productcard]") @@ -21,4 +18,3 @@ for element in elements: print("------------------ >>>") print("* " + element.text) - sb.sleep(2) diff --git a/examples/cdp_mode/raw_hacker_news.py b/examples/cdp_mode/raw_hacker_news.py index 12ababe92ff..f438bd24ac7 100644 --- a/examples/cdp_mode/raw_hacker_news.py +++ b/examples/cdp_mode/raw_hacker_news.py @@ -1,8 +1,8 @@ from seleniumbase import SB -with SB(uc=True) as sb: +with SB(uc=True, fast=True) as sb: sb.activate_cdp_mode() sb.goto("https://news.ycombinator.com/submitted?id=seleniumbase") elements = sb.find_elements("span.titleline > a") for element in elements: - print("* %s" % element.text) + print("* " + element.text) diff --git a/examples/cdp_mode/raw_planetmc.py b/examples/cdp_mode/raw_planetmc.py index 59bd1edfa3d..faf3401ead5 100644 --- a/examples/cdp_mode/raw_planetmc.py +++ b/examples/cdp_mode/raw_planetmc.py @@ -1,9 +1,14 @@ from seleniumbase import SB -with SB(uc=True, test=True, guest=True) as sb: +with SB(uc=True, test=True, incognito=True) as sb: sb.activate_cdp_mode() - sb.goto("www.planetminecraft.com/account/sign_in") - sb.solve_captcha() - # "LOG IN" is enabled when the CAPTCHA is bypassed - sb.wait_for_element_absent("input[disabled]") - sb.sleep(2) + sb.goto("https://www.planetminecraft.com/account") + sb.assert_element('input[name="login"]') + sb.assert_text("Sign in to Planet Minecraft") + sb.type('input[name="email"]', "test@example.com") + sb.type('input[name="password"]', "Fake_Password") + sb.click("input#autologin") # The checkbox + if sb.is_element_visible("input[disabled]"): + sb.solve_captcha() # Enables the input button + sb.assert_element_absent("input[disabled]") + sb.sleep(1.5) diff --git a/mcp_servers/pyproject.toml b/mcp_servers/pyproject.toml index 20cc6be8fbc..9f111277a69 100644 --- a/mcp_servers/pyproject.toml +++ b/mcp_servers/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "seleniumbase-mcp" -version = "1.4.1dev0" +version = "1.4.2dev0" description = "MCP server exposing SeleniumBase CDP Mode as tools for MCP clients." readme = "README.md" requires-python = ">=3.10" diff --git a/requirements.txt b/requirements.txt index 2b8ced43814..f26a8f17f13 100755 --- a/requirements.txt +++ b/requirements.txt @@ -40,7 +40,7 @@ selenium==4.50.0 cssselect>=1.5.0,<2 sortedcontainers==2.4.0 execnet==2.1.2 -iniconfig==2.3.0 +iniconfig~=2.3.1 pluggy==1.6.0 pytest==8.4.2;python_version<"3.11" pytest==9.1.1;python_version>="3.11" @@ -68,5 +68,5 @@ coverage>=7.16.2 pytest-cov>=7.1.0 flake8==7.4.1 mccabe==0.7.0 -pyflakes==4.0.2 +pyflakes==4.0.3 pycodestyle==2.15.0 diff --git a/seleniumbase/__version__.py b/seleniumbase/__version__.py index 58f007ff262..9c740b77429 100755 --- a/seleniumbase/__version__.py +++ b/seleniumbase/__version__.py @@ -1,2 +1,2 @@ # seleniumbase package -__version__ = "4.55.1" +__version__ = "4.55.2" diff --git a/server.json b/server.json index ad935e602f8..1f191ad18b4 100644 --- a/server.json +++ b/server.json @@ -3,7 +3,7 @@ "name": "io.github.seleniumbase/seleniumbase", "title": "SeleniumBase MCP", "description": "Stealthy browser automation, testing, and web-scraping via CDP Mode.", - "version": "4.55.1", + "version": "4.55.2", "repository": { "url": "https://github.com/seleniumbase/SeleniumBase", "source": "github" @@ -13,7 +13,7 @@ { "registryType": "pypi", "identifier": "seleniumbase", - "version": "4.55.1", + "version": "4.55.2", "transport": { "type": "stdio" }, diff --git a/setup.py b/setup.py index d3d902fd164..90a3135f27f 100755 --- a/setup.py +++ b/setup.py @@ -206,7 +206,7 @@ 'cssselect>=1.5.0,<2', 'sortedcontainers==2.4.0', 'execnet==2.1.2', - 'iniconfig==2.3.0', + 'iniconfig~=2.3.1', 'pluggy==1.6.0', 'pytest==8.4.2;python_version<"3.11"', 'pytest==9.1.1;python_version>="3.11"', @@ -247,7 +247,7 @@ "flake8": [ 'flake8==7.4.1', 'mccabe==0.7.0', - 'pyflakes==4.0.2', + 'pyflakes==4.0.3', 'pycodestyle==2.15.0', ], # pip install -e .[mcp]