Skip to content

[Python] Move the JupyROOT output capture and executors to Python - #23825

Open
guitargeek wants to merge 1 commit into
root-project:masterfrom
guitargeek:jupyroot
Open

guitargeek wants to merge 1 commit into
root-project:masterfrom
guitargeek:jupyroot

Conversation

@guitargeek

Copy link
Copy Markdown
Contributor

IOHandler.cxx captured the stdout and stderr of C and C++ code in notebooks by redirecting the file descriptors into pipes, and ran notebook cells through TInterpreter, reporting C++ exceptions. Both are now Python: the capture with os.dup2() and pipes, the cell execution through gInterpreter, with cppyy translating C++ exceptions.

The capture used to read the pipes only after flushing C stdio, so a cell that wrote more than the pipe holds (64 kB on macOS) hung: the writer blocked on the full pipe holding the stream lock that the flush waited for. Now a thread per stream drains its pipe as long as the capture runs. A process started during the capture can keep a pipe open, so the end of a capture waits at most a second for the rest of the output.

Other differences:

  • each IOHandler has its own capture, instead of all sharing one, which the first one to be deleted destroyed;
  • deleting an IOHandler ends its capture;
  • output that is not valid UTF-8 is decoded with replacement characters instead of raising;
  • ProcessLine() and Declare() release the GIL (set in the facade), so the ROOT kernel streams output while a cell runs.

🤖 Done with the help of AI

IOHandler.cxx captured the stdout and stderr of C and C++ code in
notebooks by redirecting the file descriptors into pipes, and ran
notebook cells through TInterpreter, reporting C++ exceptions. Both are
now Python: the capture with os.dup2() and pipes, the cell execution
through gInterpreter, with cppyy translating C++ exceptions.

The capture used to read the pipes only after flushing C stdio, so a
cell that wrote more than the pipe holds (64 kB on macOS) hung: the
writer blocked on the full pipe holding the stream lock that the flush
waited for. Now a thread per stream drains its pipe as long as the
capture runs. A process started during the capture can keep a pipe
open, so the end of a capture waits at most a second for the rest of
the output.

Other differences:
- each IOHandler has its own capture, instead of all sharing one, which
  the first one to be deleted destroyed;
- deleting an IOHandler ends its capture;
- output that is not valid UTF-8 is decoded with replacement characters
  instead of raising;
- ProcessLine() and Declare() release the GIL (set in the facade), so the
  ROOT kernel streams output while a cell runs.

🤖 Done with the help of AI
@github-actions

Copy link
Copy Markdown

Test Results

    2 files      2 suites   4h 59m 35s ⏱️
2 635 tests 2 635 ✅ 0 💤 0 ❌
4 955 runs  4 955 ✅ 0 💤 0 ❌

Results for commit 3fecbbf.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant