Skip to content

chore: Show correct test failure error context - #268

Open
hoodmane wants to merge 1 commit into
cloudflare:mainfrom
hoodmane:hoodmane/testlib-pickle
Open

hoodmane wants to merge 1 commit into
cloudflare:mainfrom
hoodmane:hoodmane/testlib-pickle

Conversation

@hoodmane

@hoodmane hoodmane commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

This changes testlib to propagate error context correctly from the tests inside workerd back out to the host pytest session. Previously, any error traceback would always show the host code that looked up the test results which is test support code and not related to the actual cause of the problem. Now it shows correct tracebacks. The techniques here are based on similar techniques in pytest-pyodide.

Using tblib + pickle we serialize exceptions. For robustness we use a custom pickler that pickles most things into strings, other than builtin types and types from tblib which we need to handle correctly.

On the unpickling side, we synthesize dummy exception classes on the fly for any exceptions that are missing. This is a simpler way to deal with the problems we've had in pytest-pyodide with tracebacks being messed up for exception classes that are only defined in the Pyodide session.

I used a fork of tblib called pyodide-tblib for the traceback serializing. It includes ionelmc/python-tblib#91 which fixes column positioning, a fix that makes multiline positions work correctly, and a fix that gets the first line of functions correct. This makes the context exactly match what pytest would show natively.

Before

self = <testlib.host.TestRESPONSE object at 0x7c1483072210>, dev_server = 'http://127.0.0.1:41021'

    def test_fn(self: Any, dev_server: str) -> None:
        results = get_suite_results(dev_server, suite)
        if isinstance(results, str):
            pytest.fail(results)
        result = results.get(test_name)
        assert result is not None, (
            f"Test {suite}::{test_name} not found in results; "
            f"available keys: {sorted(results)}"
        )
        if result["status"] == "skipped":
            pytest.skip(result.get("reason", ""))
        if result["status"] == "failed":
>           pytest.fail(result["error"])
E           Failed: JsException: RangeError: Responses may only be constructed with status codes in the range 200 to 599, inclusive.

After

    def test_tmp_sdk_failure():
        # Fails inside the vendored runtime SDK: bad status type.
>       Response("body", status="not-a-number").js_object
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

tests/web-frameworks-test/flask-tests/src/test_response.py:23: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/lib/python313.zip/contextlib.py:162: in __exit__
    ???
src/workers/utils.py:72: in _get_js_body
    yield body
src/workers/response.py:166: in __init__
    js_resp = js.Response.new(js_body, **options)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   pyodide.ffi.JsException: RangeError: Responses may only be constructed with status codes in the range 200 to 599, inclusive.

pyodideRuntime-internal:emscriptenSetup:23218: JsException

This changes testlib to propagate error context correctly from the tests inside
workerd back out to the host pytest session. Previously, any error
traceback would always show the host code that looked up the test results which
is test support code and not related to the actual cause of the problem. Now it
shows correct tracebacks. The techniques here are based on similar techniques in
pytest-pyodide.

Using tblib + pickle we serialize exceptions. For robustness we use a custom
pickler that pickles most things into strings, other than builtin types and
types from tblib which we need to handle correctly.

On the unpickling side, we synthesize dummy exception classes on the fly for any
exceptions that are missing. This is a simpler way to deal with the problems
we've had in pytest-pyodide with tracebacks being messed up for exception
classes that are only defined in the Pyodide session.

I used a fork of tblib called pyodide-tblib for the traceback serializing. It
includes ionelmc/python-tblib#91 which fixes column positioning, a fix that
makes multiline positions work correctly, and a fix that gets the first line of
functions correct. This makes the context exactly match what pytest would show
natively.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant