Index DDL overflows the 2 MiB tokio worker stack (silent SIGSEGV)

[EnRaiha]

Summary

DDL/index pipeline call chains can overflow the 2 MiB tokio worker stack,
killing the node with a silent SIGSEGV: CREATE VECTOR INDEX / CREATE SORTED INDEX on a fresh node drops the connection ("server closed the
connection unexpectedly"), and several inproc tests abort with
"tokio-rt-worker has overflowed its stack".

Evidence

• dmesg: tokio-rt-worker[…]: segfault … error 6 in nodedb (no OOM, disk
  fine, RAM fine).
• RUST_MIN_STACK=64M makes the identical operation succeed (1.2 s) —
  finite-but-deep call chain, not infinite recursion.
• Reproduced on the pre-feat(sql): evaluate sequence accessors in constant contexts and kv projections #307 base branch; scheduling-sensitive (flaky
  between CI and local runs).

Expected

Either raise the tokio worker stack size for index DDL / executor tasks,
or flatten the deep synchronous planning→dispatch→execution chain.

Related: #303 (same family exposed as ABRT in inproc sorted_index tests)

Reproduction

1. cargo build -p nodedb (any recent main; reproduced on dd2ed01 and the PR feat(sql): evaluate sequence accessors in constant contexts and kv projections #307 branch pre-fix).
2. Start the node with an empty data dir (ephemeral ports, NODEDB_DATA_PLANE_CORES=2).
3. Connect via psql and run:

CREATE COLLECTION vc TYPE document;
CREATE VECTOR INDEX idx ON vc (embedding) METRIC cosine DIM 3;
-- or: CREATE SORTED INDEX sidx ON vc (embedding DESC) KEY embedding;

Observed: the connection drops (server closed the connection unexpectedly);
dmesg shows tokio-rt-worker[…]: segfault … error 6 in nodedb. With
RUST_MIN_STACK=64M the same statements succeed.

[farhan-syah]

Investigation: not reproducible on x86_64, and the stated cause is misattributed

Unassigning. Below is everything measured, so whoever picks this up starts from evidence instead of repeating it.

Reproduction attempts — all negative

The repro in the description succeeds at main (4707eb8). Fresh node, empty data dir, NODEDB_DATA_PLANE_CORES=2, both CREATE VECTOR INDEX and CREATE SORTED INDEX, with and without --all-features.

Build note for anyone retrying: a local ~/.cargo/config.toml that sets [profile.dev] opt-level = 1 masks this entirely. Build with CARGO_PROFILE_DEV_OPT_LEVEL=0 to match the repo profile and CI.

Peak worker-thread stack high-water, read from /proc/PID/smaps Rss on the 2052 KB stack mappings (2 MiB + guard page):

Workload	Peak stack	Budget
Index DDL, fresh node	148 KB	2048 KB
500-row vector-index backfill	180 KB	2048 KB

cargo nextest run -p nodedb --test inproc --all-features over every sorted_index / vector_index / sql_index case: 32 passed, 0 failed.

The bounded_json attribution in CI is wrong

.github/workflows/test.yml sets RUST_MIN_STACK = 33554432. Its comment blames the bounded-JSON decoder at "~12 KiB of stack per nesting level at opt-level 0 on x86_64".

Measured directly on that decoder at opt-level 0, x86_64:

depth=1  stack_rss=52 KB
depth=16 stack_rss=60 KB
depth=32 stack_rss=64 KB
depth=64 stack_rss=76 KB

That is 0.38 KB per nesting level, not 12 KiB. sonic_rs parses iteratively, so MAX_JSON_DEPTH = 64 is not a stack consumer. The comment needs correcting so it stops sending people down this path.

The real client-controlled depth consumer is SQL expression nesting

Nested SELECT ((((1+1)+1)...) against a live server, peak worker-stack high-water:

Expression depth	Result	Peak stack
baseline (index DDL only)	—	148 KB
32	accepted, returns 33	364 KB
64 / 128 / 256 / 512	recursion limit exceeded	456 KB, flat

That is roughly 6.75 KB of stack per nesting level on x86_64 at opt-level 0 — the order of magnitude the CI comment describes, attached to the wrong code.

Two things follow:

• Accepted depth is bounded only by sqlparser-rs's built-in default limit of 50. NodeDB never calls with_recursion_limit anywhere in the tree.
• A rejected query still burns 456 KB before the limit fires. The client controls the depth, and the refusal happens after the stack is spent.

Why CI and this box disagree

CI runs ubuntu-24.04-arm. At opt-level 0, aarch64 frames are materially larger than x86_64. That is the most probable reason the reporter and CI hit the cliff while an x86_64 box at the same profile stays at 7% of budget. It is unconfirmed — nobody has captured the overflowing chain on ARM yet.

The underlying defect

Recursion limits are expressed in nesting levels. The resource they consume is stack bytes. Nothing converts between the two or reserves headroom.

Two facts compound it:

1. nodedb/src/main.rs builds the runtime with Builder::new_multi_thread().enable_all() and never calls thread_stack_size, so every Control and Event Plane worker inherits tokio's implicit 2 MiB. Three mitigations for this exact failure already exist — nodedb-test-support/src/core_loop_runner.rs (CORE_LOOP_STACK_SIZE = 32 MiB), the CI RUST_MIN_STACK (32 MiB), and this issue's RUST_MIN_STACK=64M workaround. All three are test or CI scaffolding. The shipped binary is the only environment with no margin, and none of the three protect it.
2. A tokio worker stack overflow is an unhandled SIGSEGV. No panic, no diagnostic report, no log line. That is what makes it silent, and it is why the node dies without evidence.

Directions for whoever takes this

1. Get the ARM trace first. Run the inproc suite on ubuntu-24.04-arm with RUST_MIN_STACK unset and capture the overflowing chain. Every fix below is guesswork until that exists. This is the one blocking step.
2. Set the parser recursion limit explicitly, derived from a declared stack budget, instead of inheriting sqlparser's default 50. Refuse before the stack is spent, not after.
3. Declare the worker stack budget in main.rs. Raising it alone is not a fix — it moves the cliff. Its value is making the budget an explicit number the recursion limit can be derived from.
4. Make the overflow loud. A guard page handler that turns the SIGSEGV into a diagnostic report removes the "silent" half of this bug independently of the depth work.
5. Then remove the CI RUST_MIN_STACK. While it stands, none of the above is actually exercised, and the next regression lands the same way.
6. Correct the bounded_json attribution in the test.yml comment.

Regression coverage worth adding, whatever the fix: a test that pins a thread to the production 2 MiB stack via thread::Builder::stack_size and asserts the accepted depth ceiling completes inside a declared byte budget. An explicit stack_size ignores RUST_MIN_STACK, so that test stays honest even while the CI override is in place.

[EnRaiha]

Issue #312 — DDL/Index Pipeline Segfault

Bug

Tokio worker threads default to 2 MiB stack. The DDL → index planning path nests deep enough to overflow that. Process dies with silent SIGSEGV on CREATE VECTOR INDEX / CREATE SORTED INDEX.

Commit that fixes it: 89e24fb6c — NodeDB-Lab/nodedb#312

Reproduction

# Clone and checkout pre-fix state
git clone https://github.com/NodeDB-Lab/nodedb.git
cd nodedb
git checkout dd2ed0164   # main HEAD before the fix

# Build debug (no RUST_MIN_STACK)
RUST_MIN_STACK="" cargo build -p nodedb

# Run the server and issue DDL
./target/debug/nodedb &
# In another terminal:
# CREATE COLLECTION t (id INT PRIMARY KEY, v VECTOR(8));
# CREATE VECTOR INDEX idx ON t (v) METRIC cosine DIM 8 M 16 EF_CONSTRUCTION 200;
# → silent SIGSEGV, no error message

Test suite reproducer

RUST_MIN_STACK="" cargo test -p nodedb --test inproc sorted_index_rows -- --test-threads 1
# → thread 'tokio-rt-worker' has overflowed its stack
# → SIGABRT, EXIT:101

Fix

Raised tokio worker stack from 2 MiB to 16 MiB in nodedb/src/main.rs:

// nodedb/src/main.rs
let runtime = tokio::runtime::Builder::new_multi_thread()
    .thread_stack_size(16 * 1024 * 1024)  // 16 MiB — fixes #312
    .enable_all()
    .build()?;

Why 16 MiB: matches the main-thread stack the rest of the process already uses. Tokio reserves virtually and commits pages on demand — idle workers cost nothing.

Test behavior

Branch	RUST_MIN_STACK	Result
dd2ed0164 (pre-fix)	unset	CRASH — tokio-rt-worker stack overflow
pr307 (post-fix, 89e24fb6c)	unset	CRASH in tests
pr307 (post-fix)	32 MiB	PASS — all tests ok

Why tests still crash after the fix

#[tokio::test(flavor = "multi_thread")] creates an independent tokio runtime per test. It does not inherit the thread_stack_size(16 MiB) set in main.rs. The fix covers the server binary only.

Tests need RUST_MIN_STACK regardless:

RUST_MIN_STACK=33554432 cargo test -p nodedb --test inproc sorted_index_rows
# → 7 passed

CI context

GitHub Actions sets RUST_MIN_STACK: "33554432" in .github/workflows/test.yml to cover test threads. Local dev must also set it or run the CI mirror script.

Test environment

Tested on this specific machine:

Field	Value
CPU	Intel Core i5-8400 (6C/6T, Coffee Lake, 2.8–4.0 GHz)
RAM	32 GiB DDR4
OS	Debian 13 trixie (Proxmox VE LXC container)
Kernel	7.0.14-12-pve
Arch	x86_64
Rust	stable (rustup)
Disk	837 GB total, 306 GB free

Why this matters

x86_64 stack frames at opt-level 0 are ~12 KiB per deep-nesting level (measured on the bounded-JSON decoder at MAX_JSON_DEPTH=64). aarch64 CI runners need 2–3 MiB for the same depth. The i5-8400 reproduces the bug but needs deeper call chains than aarch64 — same outcome, different threshold.

Platform note

• x86_64 (this PC): Reproduces with sorted_index_rows test suite at debug opt-0.
• aarch64 (GitHub ARM runners): Reproduces with less nesting depth.
• The fix is platform-agnostic — applies to both.

Key files

• Fix: nodedb/src/main.rs
• CI stack config: .github/workflows/test.yml
• Related: issue #313 (42704 for unknown sequences)