Semantic index cold builds are serial and lossy: no concurrent embedding, no mid-build persistence

[trim21]

Problem

Cold semantic index builds for remote embedding backends are far too slow for medium/large repos, and an interrupted build loses all progress.

1. Embedding batches are strictly serial

build_from_chunks (crates/aft/src/semantic_index.rs:2555-2572) calls the sync embed_fn(batch_texts) one batch at a time — no in-flight concurrency, and the FnMut signature rules out pipelining without a change.

Measured in the wild (OpenRouter, qwen/qwen3-embedding-8b, batch=64): ~5 chunks/s over a 34,736-chunk repo (543 batches), i.e. a ~2 hour cold build. At this batch size the bottleneck is per-request RTT, not server compute — exactly what concurrency would absorb.

There is also no config knob: the semantic config surface is only backend / model / base_url / timeout_ms / query_timeout_ms / max_batch_size / max_files (crates/aft/src/config.rs:134-158). When the upstream provider caps batch size (e.g. DashScope-backed routes reject input arrays > 20), max_batch_size: 20 multiplies the request count ~3x and makes the serial latency problem worse.

2. No mid-build persistence

semantic.bin is written only when the build completes. A cold build killed halfway (user closes the harness, machine restarts) restarts from zero — the 2-hour build has no checkpoint.

Proposal

1. Add semantic.max_concurrent_requests (default 1 = current behavior) and pipeline N batches in flight through the embed engine.
2. Periodically persist partial build progress (per-batch checkpoint keyed by the existing fingerprint), so an interrupted cold build resumes from the last flush instead of restarting.
3. Optionally: a small request coalescer/buffer so a batch-size clamp from provider errors doesn't inflate request count.

Happy to take a stab at a PR if the overall approach sounds right.

[aft-alfonso]

Confirmed on all three counts — build_from_chunks is strictly serial through the sync embed_fn, semantic.bin persists only on completion, and a provider batch clamp multiplies serial latency instead of widening in-flight work. Your ~5 chunks/s / 2-hour figure for a 34k-chunk repo is consistent with what we'd expect from that shape against a remote backend.

A PR would be very welcome. Shape constraints so it lands cleanly:

1. Concurrency: semantic.max_concurrent_requests (default 1 = today's behavior) with bounded in-flight batches is the right knob. Results must map back to chunks deterministically regardless of completion order, per-batch retry must stay per-batch (one failed batch can't lose the run), and memory stays bounded (don't buffer the whole corpus's embeddings out of order — a bounded reorder window is fine).
2. Mid-build persistence: checkpoint keyed by the existing content fingerprint, written atomically (temp + rename), resumed only on exact fingerprint match. We shipped the same design for callgraph cold builds recently (bounded resumable staging, adopt-on-match/restart-on-drift in crates/aft/src/callgraph_store/) — mirroring that adopt/restart contract keeps the two subsystems consistent.
3. Config additions need the schema artifact + cross-language parity fixtures regenerated (scripts/build-schema.ts, bun scripts/capture-config-parity.ts) — CI enforces both. Docs for the new key go in docs/config.md; we'll handle the governed-manifest alignment on merge, just call it out.
4. Interactions to preserve: the cold-build limiter and the rebind herd limiter serialize builds, not batches — in-flight concurrency lives inside one build's budget, so neither limiter contract changes.

Happy to review early if you want to open it as a draft.

[aft-alfonso]

Update on direction: we are taking this one in-house rather than inviting a PR. The three findings (serial embedding through the sync embed path, persistence only on completion, batch clamp multiplying serial latency) touch the same durability and supersession invariants the callgraph resumable build went through, so it gets a proper design pass with adversarial review rounds before code: bounded concurrent embedding per backend (the concurrency that helps a remote provider hurts a single local GPU, so it has to be backend-aware), staged persistence so a restart resumes from the last committed chunk range instead of re-embedding, and the existing batch-boundary supersession kept intact. I'll post the design summary here when it lands and link the shipped release. Thanks for the precise mechanism trace - it is the spec's starting point.