feat(index): bound indexing resources

[liuchong]

What does this PR do?

Adds one immutable, configurable resource policy for ordinary indexing requests and enforces it across discovery, supervised workers, and persistence.

• rejects dangerous exact repository roots and supports operator-defined aggregate roots
• bounds files, directories, entries, depth, source bytes, and discovery time
• caps worker CPU parallelism, physical concurrency, memory, temporary output, and duration
• checks database, staging, cache admission, and free disk while preserving the previous published index on failure
• returns structured resource_limit_exceeded errors
• prevents callers from weakening the trusted worker policy
• documents defaults and keeps cross-repo-intelligence on its existing database-only path

Related to #1347

Validation

• make -f Makefile.cbm test: 6791 passed, 4 skipped
• Focused CLI suite: 257 passed
• Focused ASan/UBSan suites: 640 passed, 3 platform skips
• make -f Makefile.cbm lint-ci: passed
• Production binary build: passed
• config --help: verified all resource keys

Checklist

• Every commit is signed off (git commit -s)
• Tests pass locally (make -f Makefile.cbm test)
• Lint passes (make -f Makefile.cbm lint-ci)
• New behavior is covered by tests

[DeusData]

Reviewed in full — every limit traced to its enforcement point and the defaults cross-checked against our recorded benchmarks. The mechanics are strong and the concern is entirely about defaults and surface size. Those are separate, and I do not want the second to overshadow the first.

What is genuinely good. The trust boundary is the best part of this PR: caller-supplied _cbm_index_limits is stripped and replaced with the server's own config policy before the worker spawns, a worker refuses a missing or incomplete policy, and there is a dedicated test that passes {"index_max_files":"999999"} and asserts the config limit still wins. test_worker_watchdog.sh then pins the full-policy contract so adding a key without updating the worker contract fails CI. That is how a supervised-worker boundary should be built.

The discovery semantics are also exactly right for this project: a limit violation aborts the request rather than returning a truncated file list — a partial index is never published as a complete one. Graph-quality-first is the rule here, and you landed on the correct side of it without being told.

Where it cannot ship as-is: the defaults contradict this project's own reference workload.

Default	Our recorded reality	Effect
index_memory_limit_mb 8192, worker killed on breach	linux-kernel index peaks 16.4–33.6 GB RSS	our flagship benchmark becomes impossible by default
index_cpu_cores 4	src/foundation/system_info.c:300 — "Use all cores for initial indexing — user is waiting"	multi-fold index-time regression for everyone with >4 cores
index_max_files 100000	our own bench is ~94,500 files at 1×	1× squeaks through at 94.5%; 2× hard-fails
index_max_file_mb 64	src/foundation/limits.c:11-14 sets a deliberate 512 MiB cap, documented as "generous"	an 8× tightening; files that index today start being skipped
index_scan_timeout_seconds 30	no discovery timeout today	cold-cache, NFS and monorepo discovery can now fail the whole request

A resource policy that kills our own reference index out of the box is not a shape we can adopt. If the direction is approved, the version I would argue for is unlimited (off) by default, with the policy opt-in — the enforcement machinery keeps all its value and nothing regresses for existing users.

Two more things worth surfacing louder:

1. cbm_pipeline_new now bakes default limits into every pipeline, so watcher background reindexes are bound too. A repo that grows past a cap after initial indexing has its refreshes fail into logs and index_status only — the user quietly keeps a stale graph without an obvious signal. Worth making noisy, e.g. a stale-index warning on the next MCP response.
2. 18 permanent config keys is a very large one-way door for one feature. If this is adopted, collapsing to something like one index_resource_profile plus a few targeted overrides would age much better.

Security is clean: no new dependencies, no CI or workflow changes, no prompt-injection surface, and the two raw fopen calls are Linux-only /proc/<pid>/stat reads behind #elif defined(__linux__), which is fine — fixed ASCII procfs paths do not need cbm_fopen. New file writes use cbm_fopen correctly. The root-safety hardening (refusing /, $HOME and the cache dir as exact canonical roots while leaving descendants indexable) is a sensible touch.

Whether this project wants a resource-policy subsystem at all is the maintainer's call, not mine, so I have escalated it with the above attached rather than deciding in review. You will also need a rebase — the PR is CONFLICTING and touches mcp.c, pipeline.c and cli.c, all of which are moving.

Thank you for the care in this one; the parts that are good are good for the right reasons.

[liuchong]

Thanks for the thorough review. Agreed on the defaults and surface concerns. I'll hold off on reshaping and rebasing until the policy direction is clear. ❤️

[DeusData]

Thank you for waiting while we settled the product direction. The concept is worth pursuing, specifically as an opt-in safety and resource-control surface. We are not ready to change defaults in ways that could regress established large-repository workloads.

To make the work reviewable, please split it into atomic slices:

1. Exact-root safety
2. Discovery-side limits
3. Worker watchdog limits
4. Storage, cache, and free-disk admission

For the public configuration surface, we would prefer a compact profile with a small number of justified overrides rather than eighteen independent keys. Each slice should preserve current defaults, include deterministic tests, and explain the operational failure mode it controls. This approves further work in that opt-in shape, but it is not yet a merge commitment for a particular API. Thank you for the thoughtful trust-boundary analysis and for giving us room to establish a maintainable direction.

[liuchong]

Thank you for the direction. I have rebuilt the work in the opt-in shape you asked for, as a stack of six small PRs rather than a reshaped version of this branch. Opening them fresh gives each slice its own review thread and a clean diff against current main; this branch has drifted far enough from what I am now proposing that reusing it would make review harder, not easier.

The stack

#	PR	Slice
1	#1723	Discovery-side limits
2	#1724	Worker watchdog limits
3	#1725	Storage, cache and free-disk admission
4	#1726	index_resource_profile
5	#1727	Attempt record and Git-backed freshness
6	#1728	Stale-index warning on the next MCP response

PRs 1–3 are your slices 2–4. PR 4 is the compact profile you asked for. PRs 5 and 6 answer point 1 of your first review — the watcher rebuild that fails into the logs and leaves the user on a stale graph with no obvious signal. PR 1 is ready for review; the rest are drafts until the one below it is settled, so you are never asked to review two moving diffs at once.

An honest note about the split. You asked for four slices and I am handing you six, plus six review threads instead of one. Splitting also has real costs here: because a fork PR has to target a branch in this repository, each PR's diff carries the commits under it until the one below merges, so the later diffs look larger than the work they add. And six threads means six CI runs and six places for the conversation to fragment.

If you would rather have one PR, say so and I will drop the stack and make the same changes on this branch instead — same content, same defaults, one thread. I split it because you asked for atomic slices and I think each of these is genuinely separately decidable, but the shape is your call, not mine, and I would rather ask than make you review six things you did not want.

Defaults

Everything defaults to off. With no configuration, none of the six PRs changes any behaviour, so the recorded large-repository benchmarks are unaffected. The balanced profile derives its worker memory ceiling from detected host memory rather than a fixed number, precisely because of the 16.4–33.6 GB figure you cited: any round ceiling low enough to look safe would reject an index that succeeds today.

Slice 1, exact-root safety, is not in the stack

src/foundation/workspace.c now classifies and refuses exact roots — home directory, credential directories, shallow top-level trees, Windows system trees — with the cache-containing-root case deliberately left indexable and documented as such. That is the same protection my branch was adding, done better and already on main, so re-landing it would only create a conflict.

Three capabilities from this branch are dropped on purpose

• index_max_file_mb — an 8× tightening of the documented 512 MiB cap in src/foundation/limits.c. Files that index today would start being skipped, which is a breaking change and belongs in its own issue rather than riding along with an opt-in policy.
• index_cpu_cores — contradicts the deliberate "use all cores for initial indexing — the user is waiting" decision in src/foundation/system_info.c. Capping parallelism from a config key fights the existing concurrency model instead of extending it.
• index_low_priority — same conflict, and my original implementation had no test evidence that it changed anything measurable. I would rather drop it than ship an unverified knob.

The public surface is now index_resource_profile plus six individual keys, all off by default, down from eighteen. The remaining dimensions — directory count, entry count, depth, discovery deadline, and the three storage internals — exist only inside a profile, because they are meaningful as part of a composed decision and not as standalone operator settings.

I have not touched this branch or this PR. Close it in favour of the stack, leave it open as context, or tell me to bring the work back here and close the six — whichever costs you least.