Looking for feedback: adds interceptor lifecycle hook

[skrawcz]

This proposes a new hook to allow one to remotely push execution of a Burr Action.

For example, if one wants to selectively push remote execution of an action to say Ray,
then one can now build an interceptor.

This also introduces companion hooks that would run pre & post on the remote to mirror
what Burr has today - -I thought it would be simpler to not try to reuse those hooks, and
instead make it explicit as to what is going on if you wanted to add something similar.

See tests for examples, but open to feedback here.

Note: we'd need to think through how this plays with parallelism functionality we have.
E.g. currently interceptors aren't propagated to sub applications...

Changes

• adds to internal hooks to intercept an action and remotely execute it
• adds ray example

How I tested this

• NEEDS EXTENSIVE TESTING
• Test sync
• Test async
• Test async streaming
• Test sync streaming
• I ran some ray examples
• need to write another interceptor for another backend, e.g. temporal to feel it out.

Notes

This change needs to have some thought and discussion.

Checklist

• PR has an informative and human-readable title (this will be pulled into the release notes)
• Changes are limited to a single goal (no scope creep)
• Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future TODOs are captured in comments
• Project documentation has been updated if adding/changing functionality.

[github-actions]

A preview of 105da95 is uploaded and can be seen here:

✨ https://burr.dagworks.io/pull/602 ✨

Changes may take a few minutes to propagate. Since this is a preview of production, content with draft: true will not be rendered. The source is here: https://github.com/apache/burr/tree/gh-pages/pull/602/

[goutamk09]

@skrawcz Thanks for opening this — I really like the direction of making remote execution explicit via lifecycle hooks.

Before diving into design feedback, I’m trying to fully understand the current execution flow vs the new interception path. A few things I’m exploring:

• Where in the action execution pipeline the new hook takes over execution
• How this behaves across sync / async / streaming paths
• Why tests are failing on 3.11 and 3.12 but not 3.9 / 3.10
• How this might interact with sub-applications and parallel execution (as noted in the PR)

I’ll follow up with more concrete observations once I’ve traced the execution path locally.

[goutamk09]

Hey @skrawcz
I’ve run the interceptor-specific integration tests locally on Python 3.11 and they are passing (sync, async, streaming, worker hooks).

Running the full test suite locally shows failures related to missing optional integration dependencies (redis, mongodb, asyncpg, langchain, etc.), not interceptor behavior.

From what I can see so far, the interceptor lifecycle implementation itself appears stable under 3.11.

I’m continuing to look into what might be causing the CI 3.11/3.12 failures — especially around async execution paths or example validation.

[goutamk09]

@skrawcz I ran the full core test suite locally under Python 3.11.

All execution, lifecycle, async, and persistence tests pass (264 passing).

The only failure is in test_graphviz_display, which requires the Graphviz system binary (dot) to be installed and available on PATH. This appears to be an environment/dependency issue rather than a behavioral regression in the interceptor or lifecycle implementation.

Given that all core execution paths pass under 3.11, I don’t see evidence of a Python-version-specific async or hook-related issue.

If CI failures are similar, it may be worth ensuring the Graphviz binary is installed in the CI environment or marking visualization tests as optional.

tests\core\test_state.py::test_register_field_serde_check_no_kwargs PASSED              [ 98%] 
tests\core\test_state.py::test_state_apply_keeps_typing_system PASSED                   [ 98%] 
tests\core\test_state.py::test_state_keys_returns_list PASSED                           [ 99%] 
tests\core\test_validation.py::test__assert_set PASSED                                  [ 99%] 
tests\core\test_validation.py::test__assert_set_unset PASSED                            [100%]

============================= 264 passed, 5 deselected in 8.63s ============================== 

[goutamk09]

Hey @skrawcz — quick update from my side:

I ran core tests locally on Python 3.11 and they pass (264 passed; only optional Graphviz dot env issue can fail if binary missing).

CI is currently failing for py3.11/py3.12 + validate-examples, but the logs expired, so we can’t see root cause.

Could you please re-run the failed jobs (or the full workflow) so we get fresh logs? Once logs are available, I’ll triage and propose a fix PR (workflow/env vs code).

[goutamk09]

@skrawcz — local test update (Windows, Python 3.12):

• Running python -m pytest -q from the repo root fails during collection because tests under examples/ expect local JSON fixtures (e.g. hypotheses_test_cases.json, prompt_for_more.json), and there’s also an import mismatch due to the duplicate test_application.py name in examples/ vs tests/.
• Running the main test suite with python -m pytest tests -q works after installing Graphviz (both the Python package and the dot system binary).
• Current local result: 444 passed, 3 skipped.

I did notice one PytestUnraisableExceptionWarning related to SQLitePersister.del, but it does not fail the suite.

If there are specific CI failures you’d like me to focus on, I’m happy to investigate further.

[skrawcz]

@goutamk09 sorry missed all these. will try to find some time this week to respond -- feel free to use the dev@ apache email list next time to discuss ...

[goutamk09]

Thanks @skrawcz — no worries at all. I’ll use the dev@ list next time for broader discussion.
I’ll keep an eye on the CI failures once logs are refreshed and happy to help if needed.