feat: let callers opt out of index creation with create_index=False

[vishal-bala]

Stacked on #685, which fixes the connection-time failure this PR's credential also hits. Review that one first.

Every extension checks whether its index exists while being constructed, and that check is FT.INFO. A credential assembled from +@read +@write is denied FT.INFO and FT.CREATE together — neither command is in either category, identically on Redis 8.0.6 through 8.8.1 — so such a role cannot construct SemanticCache, MessageHistory, SemanticMessageHistory or SemanticRouter at all, even against an index it can query perfectly well:

RedisSearchError: Error while fetching llmcache index info:
User <name> has no permissions to run the 'FT.INFO' command

There was no way to ask for less. overwrite=False is the reason the check runs — create() calls exists() first and consults overwrite only afterwards — drop is not a constructor parameter at all, and overwrite=True is strictly worse, since it proceeds to FT.DROPINDEX. SearchIndex exposes no lifecycle seam either, and each constructor calls create() inline with no hook to subclass around.

create_index=False lets the caller state what RedisVL cannot ask: the index exists. It skips the existence check, the schema comparison against the live index, and creation, so the constructor issues no index command at all. It is rejected together with overwrite=True, which asks for the opposite.

Why not a runtime probe

FT.SEARCH is permitted where FT.INFO is not, so probing with it looks attractive. Its reply is identical for an index and for an alias pointing at one, though, so a probe would reinstate the create(overwrite=True, drop=True) → FT.DROPINDEX <alias> DD data-loss path that #672 closed. And a credential that cannot ask whether the index exists cannot create one either, so there is nothing to work out at runtime — exists() keeps failing loudly instead.

The invariant this had to fix first

Constructor-time create() was the de-facto eager connect. SearchIndex.client returns the raw client and is None until the lazy _redis_client property runs, so skipping create() left ten self._index.client sites in redisvl/extensions/ dereferencing None — starting with the router's route_config write, which runs immediately after index setup. All ten now use _redis_client. Two of the # type: ignore comments they carried turned out to cover a real scan_by_pattern signature mismatch rather than the Optional, and are kept with explicit codes.

Router semantics

create_index=False means the index exists, is already seeded, and is not ours to rewrite, so the router also skips writing route references and the stored route_config. Rewriting that blob from an unverified local route list would truncate a shared router's routes, and JSON.SET is @write, so a restricted credential can do it. _update_router_state() stays armed — add_route() and remove_route() are the caller acting deliberately — but that consequence is now stated in add_route()'s own docstring as well as the guide, since the docstring is the reference for anyone who never opens the guide.

SemanticRouter.from_existing() threads the flag through. It reads the stored config with JSON.GET and reaches FT.INFO only via the constructor, so with create_index=False it issues no index command and becomes the way to attach to a router under a restricted credential. The flag is popped before _split_from_existing_kwargs, which retains only SearchIndex init kwargs and would otherwise pass it to the Redis client constructor — where SearchIndex.__init__ discards unknown kwargs silently, making the mistake invisible.

Separately, routes=[] now raises a useful error when matching instead of max() arg is an empty sequence. That is unconditional: emptiness is legal on either path, and a flag about index ownership should not decide it.

Tests

tests/unit/test_extension_create_index_flag.py — 17 cases. The contract is "no index command at all", so they assert on the client: a MagicMock records every call, and ft() is the gate every FT.* command passes through. All four constructors reach zero recorded calls.

Two cases pin the lazy-connect invariant above by driving drop(id=...) and get_route_references() as the first operation — reverting any of the ten conversions otherwise breaks no test. Others pin that the flag survives as instance state, that it never reaches the router's stored config, and that from_existing() still verifies by default.

The default construction path is not re-tested here; the existing 189 integration tests already fail if create() is dropped.

One integration test round-trips store()/check() through a cache built with create_index=False under a real +@read +@write -@dangerous ACL user — the customer's rule, so the destructive commands it denies stay denied — with the premise pinned (FT.INFO must raise NoPermissionError for that user) and the negative alongside it: without the flag the same credential raises RedisSearchError naming ft.info, with NoPermissionError chained.

Docs

The ACL section of docs/user_guide/installation.md is restructured. Four statements were falsified by this change or were already wrong: that a credential needs @search at all, that all four extensions always call create(), that enumeration is the only thing an -@admin rule breaks, and the advice to grant FT.CREATE. The operation table gains a +@read +@write column, the command-to-category mapping is labelled as measured rather than documented, and the wrapped error text appears verbatim under its own heading — an H3, so it has an anchor to link to.

The pre-existing key-permission material became its own section rather than being dropped, and was corrected while there: partial key-pattern overlap is denied exactly like no overlap, not filtered down to the readable subset, and FT.CREATE is not key-checked at all, so a credential can create an index it cannot query.

A new subsection covers what the flag gives up. An absent index fails loudly, and a vector dimension mismatch does too — but only once the index holds a document, which a freshly provisioned index will not. A wrong prefix, an ON JSON index written as hashes, and a differing datatype or distance metric are silent. The tell is FT.INFO's key_type, prefixes and attributes, not hash_indexing_failures, which stays 0 because those keys were never indexing candidates. Router provisioning gets its own subsection, since preparing one for this mode needs embedded reference vectors rather than a hand-written FT.CREATE.

Two corrections worth calling out for reviewers who know this area: clear() is not uniform — only SemanticCache.clear() avoids FT.INFO, while the other three delegate to SearchIndex.clear(), which calls info() first — and Redis Cloud's predefined Read-Write rule reads as @read/@write-shaped from its published description, so it is a candidate for this problem rather than immune to it.

Not in scope

create_index=False restrains construction only; delete() and clear() stay armed. Coupling ownership to the flag is the coherent next step, and the flag is stored as instance state so it can be added without another parameter.

adk_redis needs a matching field on RedisVLCacheProviderConfig before this reaches callers who construct through that provider. Until then the interim for the escalation remains the exists() monkeypatch already shared in the thread.