Spread indexing pipelines behind an environment variable

[loutPhilipps]

Make it possible to spread indexing pipelines over the commit timeout duration by enabling an environment variable.

This reuses the cooperative indexing logic: the hashed pipeline id is used to determine a target phase for a given pipeline, and every subsequent timeout message is sent to the inbox with a shift to nudge the phase towards it ideal phase (with a maximum of 5s each time).

Note

I needed to be able to shift the actor's time manually, this is why shift_time was introduced on the actor's universe. I am not very confident in this change, but I didn't manage to test this PR's logic without it. Feel free to suggest a better idea.

Motivation

On large indexers, we notice that indexing pipelines can become synchronized. Under load, commits are driven by the commit timeout, and since all indexing pipelines start at the same time, they compete for the same resource (CPU/IO) at the same time.

Testing

Tested with 4 indexers with 31 vCPUs, with enough volume to have high usage. The impact of deploying this change is modest but it helps smooth out CPU usage:

Although Datadog metrics might not have a fine enough resolution to see the impact in details, it seems that enabling this does improve the resource usage.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-15T11:53:12.085180Z	78113ee	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

https://github.com/quickwit-oss/quickwit/blob/78113ee85dff3f799f09825ea01fa4a5b3d1d151/quickwit-indexing/src/actors/indexer.rs#L68
Document the spread-indexing environment variable

Add operator-facing documentation for QW_ENABLE_SPREAD_INDEXING_PIPELINES, including its default, its precedence relative to cooperative indexing, and that enabling it can extend a configured commit_timeout_secs by up to five seconds. Without this, operators cannot discover or safely assess the new runtime configuration, while existing documentation still describes the commit timeout as a strict maximum. The repository explicitly requires documentation updates for public behavior and configuration changes.

AGENTS.md reference: AGENTS.md:L23-L24

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".