Skip to content

[C++] Hang in ThreadPool::Shutdown() after failure to spawn a worker thread #48137

New issue
New issue
Closed
Closed
[C++] Hang in ThreadPool::Shutdown() after failure to spawn a worker thread#48137
Assignees
advitrocks9
Labels
Component: C++Type: bug
Milestone
26.0.0

Description

@Pear0
Pear0
opened on Nov 14, 2025

Describe the bug, including details regarding any error messages, version, and platform.

I have a case where my C++ application that uses arrow hangs on shutdown after failing to construct a thread for the IOThreadPool.

Expected behavior: clean shutdown after catching the exception or some kind of fatal condition
Actual behavior: hang on shutdown after catching the exception
Platform: Linux x86_64
Arrow Version: 19.0.1 (but based on reading code I believe this to still be an issue on main)

It is pretty easy to trigger std::thread::thread() to throw by setting RLIMIT_NPROC to something extreme like 1.

If std::thread::thread() throws in ThreadPool::LaunchWorkersUnlocked() there is a state_->workers entry even though the thread was never spawned. It is the thread's responsibility to remove the list entry when it exits so ThreadPool::Shutdown() never reaches its state_->workers_.empty() condition.

This is usually not noticeable because an abnormal shutdown will not call the IOThreadPool's destructor. In my case, I have a C++ binary that catches the exception then exits "normally" with an error message and exit code. This does call the ThreadPool destructor leading to the hang.

Hopefully this is clear. If it is not, I could put together a repro example.

Component(s)

C++

Activity

  1. Pear0
    added
    Type: bug
    on Nov 14, 2025
  2. github-actions
    added
    Component: C++
    on Nov 14, 2025
  3. advitrocks9
    added a commit that references this issue on Sep 2, 2026

    apacheGH-48137: [C++] Return a Status when a worker thread fails to s…

    0fa803a
  4. pitrou
    added a commit that references this issue on Sep 3, 2026

    GH-48137: [C++] Restore ThreadPool state when a worker fails to start (…

    157575a
  5. pitrou
    added this to the 26.0.0 milestone on Sep 3, 2026

  6. pitrou commented on Sep 3, 2026

    @pitrou
    pitrou
    on Sep 3, 2026
    Member

    Issue resolved by pull request 51107
    #51107

  7. pitrou
    closed this as completedon Sep 3, 2026

  8. pitrou commented on Sep 3, 2026

    @pitrou
    pitrou
    on Sep 3, 2026
    Member

    @advitrocks9 Can you post here so that we assign this issue to you?

  9. advitrocks9 commented on Sep 3, 2026

    @advitrocks9
    advitrocks9
    on Sep 3, 2026
    Contributor

    Fixed in #51107

  10. pitrou
    assigned
    advitrocks9
    on Sep 3, 2026
  11. whr778
    added a commit that references this issue on Sep 30, 2026

    fetch_fineweb: exit without C++ teardown; pyarrow 25.0.1 deadlocks th…

    1e6f33b

  12. whr778 commented on Sep 30, 2026

    @whr778
    whr778
    on Sep 30, 2026

    We hit this on pyarrow 25.0.1 (macOS arm64, Python 3.12) through Hugging Face datasets streaming parquet: ~176 configs read concurrently from 8 Python threads. Every file is written and main() returns, but the process never exits.

    A native sample of the stuck process shows:

    exit (libsystem_c)
  __cxa_finalize_ranges
    std::shared_ptr<arrow::internal::ThreadPool>::~shared_ptr()
      arrow::internal::ThreadPool::~ThreadPool()
        arrow::internal::ThreadPool::Shutdown(bool)
          std::condition_variable::wait

    with 8 pool workers (ThreadPool::LaunchWorkersUnlocked threads) idle in condition_variable::wait, at 0% CPU, indefinitely. It reproduced 3/3 on 25.0.1 and does not reproduce on 26.0.0.dev323, which includes GH-48137's fix. We could not bisect, since every available nightly postdates the fix, so this is consistent with rather than proven to be GH-48137.

    Would a backport to a 25.0.x patch release be possible? Many datasets users on 25.x would hit this at the end of long streaming jobs.

  13. pitrou commented on Sep 30, 2026

    @pitrou
    pitrou
    on Sep 30, 2026
    Member

    Hi @whr778 , I think it's unlikely that we'll make a 25.0.x patch release since we plan to release 26.0.0 in the next month. @raulcd might elaborate.

  14. raulcd commented on Sep 30, 2026

    @raulcd
    raulcd
    on Sep 30, 2026
    Member

    The feature freeze for 26.0.0 is scheduled for tomorrow. This will be part of it.
    Currently due to our extensive CI heavy release process we do not backport patches to previous releases unless the main release is still far away.

    More details on why we don't do it: We require many third party libraries and CI jobs to build correctly in order to be able to release and it is not only backporting this commit but the many commits that have been done during the last 3 months to keep CI green to fix the various jobs and dependency issues. Honestly we do not have the resources to do that at the moment so we release bug fixes on the new major release.

    The only issues that would grant a patch release like that would be critical security fixes.

  15. dashdartgit commented on Nov 5, 2026

    @dashdartgit
    dashdartgit
    on Nov 5, 2026

    Following up on the 2026-09-30 report above: we still see this stack on pyarrow 26.0.1 (released 2026-11-03), macOS 27 arm64, CPython 3.12, pandas 3.0.3.

    A script that reads three small local parquet files with pd.read_parquet(), prints one line and returns from main() hung at exit in 11 of 1,000 runs on 26.0.1. Every hang, sampled with sample, shows the stack above (exit → __cxa_finalize_ranges → ~shared_ptr<ThreadPool> → ThreadPool::Shutdown → condition_variable::wait), with an idle pool worker in condition_variable::wait and no PyGILState_Ensure frame, so as far as we can tell it is not #50482's mechanism.

    It needed all three of:

    1. the Arrow CPU pool used (here, pd.read_parquet on local files);
    2. stdout connected to a pipe (subprocess.run(..., capture_output=True)): 14 hangs in 1,500 runs on a pipe, against 0 in 1,500 with stdout redirected to a file and 0 in 600 on a tty;
    3. the process exiting within tens of milliseconds of its last read. Each of these gave 0 in 600, beside about 10 in 600 for the unmodified script: a 50 ms sleep before exit, pa.set_cpu_count(1), OMP_NUM_THREADS=1, pq.read_table(path).to_pandas(), or pd.read_parquet(path, filesystem=pyarrow.fs.LocalFileSystem()).

    Other versions, same harness (6 runs in parallel; a run counts as hung when it is still alive 25 s after printing its last line): 21.0.0 15/800, 23.0.1 8/800, 24.0.0 about 1.7–2.5%, 26.0.1 11/1,000.

    Minimal reproducer (16 hangs in 600 runs on 24.0.0, same harness):

    # setup.py — run once
import os
import pandas as pd
os.makedirs("data", exist_ok=True)
for k in range(3):
    pd.DataFrame({"ticker": ["AAA"] * 2, "date": ["2023-01-03", "2023-01-04"]}).to_parquet(f"data/f{k}.parquet")
    # repro.py — run many times, e.g.
#   subprocess.run([sys.executable, "repro.py", "data"], capture_output=True, timeout=25)
import sys, glob
import pandas as pd

for f in sorted(glob.glob(sys.argv[1] + "/*.parquet")):
    pd.read_parquet(f)
print("DONE")

    Happy to open a separate issue if that is more useful than a comment on a closed one.

  16. raulcd commented on Nov 5, 2026

    @raulcd
    raulcd
    on Nov 5, 2026
    Member

    @dashdartgit we should open a new issue to track your failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Component: C++Type: bug

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions