Skip to content

fix(bootstrap): raise tokio worker stack to 16 MiB - #320

Merged
farhan-syah merged 1 commit into
mainfrom
fix/issue312-stack-overflow
Sep 11, 2026
Merged

fix(bootstrap): raise tokio worker stack to 16 MiB#320
farhan-syah merged 1 commit into
mainfrom
fix/issue312-stack-overflow

Conversation

@EnRaiha

@EnRaiha EnRaiha commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Problem

Tokio worker threads use a 2 MiB default stack. The DDL → index planning path nests deep enough to overflow it, causing a silent SIGSEGV with no error message. Affected operations:

  • `CREATE VECTOR INDEX`
  • `CREATE SORTED INDEX`

The process dies mid-planning with `tokio-rt-worker has overflowed its stack` (or silently on the server wire path).

Reproduction

git clone https://github.com/NodeDB-Lab/nodedb.git
cd nodedb
RUST_MIN_STACK=\"\" cargo test -p nodedb --test inproc sorted_index_rows -- --test-threads 1
# → thread 'tokio-rt-worker' has overflowed its stack
# → SIGABRT, EXIT:101

Tested on x86_64 (Intel Core i5-8400, 32 GiB RAM, Debian 13 trixie / Proxmox VE). aarch64 CI runners reproduce with less nesting depth.

Fix

Raise the tokio worker stack from 2 MiB to 16 MiB in `nodedb/src/main.rs`. This matches the main-thread stack the rest of the process already relies on. Tokio reserves the space virtually and commits pages on demand, so idle workers cost nothing.

let runtime = tokio::runtime::Builder::new_multi_thread()
    .thread_stack_size(16 * 1024 * 1024)  // 16 MiB — fixes #312
    .enable_all()
    .build()?;

Verification

Config Result
Pre-fix + no `RUST_MIN_STACK` CRASH — stack overflow
Post-fix + no `RUST_MIN_STACK` PASS (server binary)
Post-fix + `RUST_MIN_STACK=32MiB` PASS (16/16 inproc tests)

Note: `#[tokio::test]` creates an independent runtime per test, so the test suite still requires `RUST_MIN_STACK=33554432` (already set in CI). The fix covers the server binary, which is what runs in production.

Files changed

  • `nodedb/src/main.rs` — add `thread_stack_size(16 MiB)` to the tokio runtime builder

Copilot AI lite review requested due to automatic review settings September 10, 2026 14:47

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@EnRaiha EnRaiha added the run-ci Opt this PR into the full test suite; re-add to force a re-run label Sep 10, 2026
The worker threads carry the whole planning -> dispatch -> execution
pipeline synchronously; the DDL/index path nests deep enough to exhaust
tokio's 2 MiB default and the process died with a silent SIGSEGV
(CREATE VECTOR/SORTED INDEX dropped the connection; inproc tests aborted
with 'tokio-rt-worker has overflowed its stack'). 16 MiB restores
headroom; tokio reserves the space virtually and commits pages on
demand. Verified: index DDL and the previously-aborting suites pass
without RUST_MIN_STACK.
@EnRaiha
EnRaiha force-pushed the fix/issue312-stack-overflow branch from c0a65ac to efa0277 Compare September 10, 2026 22:13
Comment thread nodedb/src/main.rs Outdated
@farhan-syah
farhan-syah merged commit a4d6ac7 into main Sep 11, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci Opt this PR into the full test suite; re-add to force a re-run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants