Skip to content

Graphs cannot run inside an existing event loop (Jupyter, Colab, FastAPI), still in 2.1.6 #1116

Description

@hakoke

Running a graph from code that already has an event loop still fails on 2.1.6 with the error reported in #122, #179, #186, #694, #708 and #771:

RuntimeError: asyncio.run() cannot be called from a running event loop

Minimal repro, no LLM or network needed, the error fires before any fetch:

import asyncio
from scrapegraphai.docloaders.chromium import ChromiumLoader

async def main():
    loader = ChromiumLoader(["https://example.com"], headless=True)
    return loader.load()

asyncio.run(main())

The same call works at the top level of a plain script. The crash comes from ChromiumLoader.lazy_load, which calls asyncio.run() per URL (scrapegraphai/docloaders/chromium.py line 463 on current main). FetchNode uses loader.load(), so any graph that fetches a URL with the default loader hits this in notebooks, FastAPI handlers, and any other code running under a loop.

Two nodes try to handle the running-loop case but the guard is inverted. graph_iterator_node.py lines 74-75 and generate_answer_from_image_node.py lines 121-123 call eventloop.run_until_complete() exactly when eventloop.is_running() is true, the one situation where that call always raises. Those branches can only ever have worked with nest_asyncio applied. And nest_asyncio is a patch on the loop rather than a fix in the library: in #179 a user who applied it just hit a different error.

The repo already has the right idea in AbstractGraph.run_safe_async, which offloads the blocking call to an executor. Applying the same idea at the four call sites that use asyncio.run or run_until_complete (chromium.py, browser_base.py and the two nodes above) fixes these environments without changing anything for plain scripts. Tested on 2.1.6 (current main), Python 3.12, Windows. I have the fix ready and will open a PR in a few minutes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions