Fixes #33572: scope search requests to the cluster alias instead of the whole cluster

[mohityadav766]

Describe your changes:

Fixes #33572

Search indexing died before its first record on every deployment whose search role is confined to its own <clusterAlias>* prefix — every shared-tenancy Collate Cloud cluster:

RuntimeException: Cannot determine the live chunk target for <clusterAlias>_data_asset_embeddings_chunks
Caused by: java.io.IOException: Forbidden access

resolveLiveChunkTargetStrict probed the chunk read alias with existsAlias(name), which builds HEAD /_alias/{name} — no index in the path. OpenSearch resolves that against _all, so it needs cluster-wide indices:admin/aliases/get, and a prefix-scoped role is denied. The 403 cannot degrade to "no alias" either: RestClientTransport.getHighLevelResponse raises TransportException("Forbidden access") on status 403 before the BooleanEndpoint status→boolean mapping runs. requireResolvedLiveChunkTarget then threw, and RecreateWithEmbeddings stages the chunk recreate before super.reCreateIndexes — so the whole run aborted, producing the failed · 0 records signature.

Introduced by #30364, which both un-gated the staged recreate (previously dead code behind a flag that could never be true) and swapped a swallowed GET /_alias/{base} for this strict probe. Before it, the same 403 was caught at debug level and resolution fell through to the index-scoped HEAD /{base}, which the role does allow.

The rule this PR enforces: every request names a concrete index or pattern beginning with the cluster alias; nothing resolves to _all.

What changed

1. OpenSearchVectorService.resolveLiveChunkTargetStrict — GET /{base}/_alias instead of HEAD /_alias/{base} + HEAD /{base}. Naming the index in the path is authorized by the same prefix-scoped role, and one call covers both layouts because the response is keyed by physical index: the staged generation when base is the read alias, base itself when it is still the legacy physical index. A 404 means neither exists yet.

2. beginStagedChunkRecreate — degrades to "not staged" on an indeterminate probe instead of throwing. Skipping the stage deletes nothing (the orphan sweep below it never runs), so an unanswerable probe is a reason to leave the chunk index alone, not to fail an entity reindex that does not need it. The promote path stays strict — that swap does remove the previous target.

3. getIndicesByAlias (OpenSearch + Elasticsearch) — used the same cluster-wide form for both the existsAlias pre-probe and GetAliasRequest.name(). Its 403 was swallowed, so it silently returned an empty index set. Now index-scoped, pre-probe dropped (the 404 branch already covers "no such alias"), and the result filtered back to indices that actually carry the alias since GET /{name}/_alias resolves the name as index-or-alias.

4. SearchRepository.indexTemplatesMatch — read om_*, returning every co-tenant's templates on a shared cluster. Scoped to om_<clusterAlias>_*.

No new cluster-wide grant is required. Notably, granting indices:admin/aliases/get on * would have "fixed" (1) at the cost of letting any tenant enumerate the others' indices, which is why the fix belongs in the request rather than the role.

Type of change:

• Bug fix

High-level design:

N/A — small change.

Tests:

Use cases covered

• A full reindex on a deployment with a prefix-scoped search role completes instead of aborting at 0 records
• A chunk probe that the cluster cannot answer skips staging and leaves existing chunks live, rather than failing the reindex
• A fresh install (no chunk index or alias) still stages its first generation
• getIndicesByAlias returns only indices that actually carry the alias
• Index-template comparison reads only this deployment's templates

Unit tests

• I added unit tests for the new/changed logic.

Files updated:

• OpenSearchVectorServiceChunkStagingTest — resolveLiveChunkTarget_asksForTheChunkIndexByName_notForAClusterWideAliasLookup asserts the captured GetAliasRequest carries index and an empty name; ..._treatsA404AsFreshInstallRatherThanAnIndeterminateProbe; beginStagedChunkRecreate_skipsStagingWhenTheLiveTargetProbeIsIndeterminate replaces the old abort-on-probe-failure test and verifies nothing in the cluster is touched
• OpenSearchIndexManagerTest / ElasticSearchIndexManagerTest — testGetIndicesByAlias_NamesTheIndexInThePathNotTheAlias, plus SuccessfulRetrieval extended with a decoy index that does not carry the alias
• IndexTemplateManagerTest — liveTemplateLookupIsScopedToThisDeploymentsClusterAlias

Result: Tests run: 2435, Failures: 0, Errors: 0 across org.openmetadata.service.search.**.

Backend integration tests

• Not applicable (no API surface change).

Ingestion integration tests

• Not applicable (no ingestion changes).

Playwright (UI) tests

• Not applicable (no UI changes).

Manual testing performed

Verified against the upstream client and server sources rather than a live restricted cluster:

• ExistsAliasRequest's endpoint emits /_alias/{name} when no index is set, and /{index}/_alias/{name} when one is — confirming the old probe named no index
• RestClientTransport.getHighLevelResponse throws TransportException("Forbidden access") on 403 before the boolean-endpoint mapping, confirming the 403 could never read as false
• GetAliasRequest's SimpleEndpoint treats 404 as an error, so the fresh-install branch arrives as OpenSearchException(404) — the same contract the pre-existing getIndicesByAlias 404 handling already relies on

UI screen recording / screenshots:

Not applicable.

Follow-ups (not in this PR)

• PUT /_search/pipeline/hybrid-rrf needs cluster:admin/search/pipeline/put, which a prefix-scoped role does not have. It is caught and warned, so it does not fail the reindex — but the pipeline is then never created and hybrid search silently degrades. The clean fix is an inline (ad-hoc) search pipeline in the request body, which needs no cluster permission and lets per-tenant weights actually differ; that change spans the query side too, so it is separate.
• SearchClusterFitnessAnalyzer's /_cat/indices and /_cat/aliases probes could be scoped to {clusterAlias}*. They already degrade gracefully via the analyzer's inaccessible set, so this is cleanup rather than a fix.
• cluster_manage_index_templates remains cluster-wide — index-template actions cannot be pattern-scoped by the security plugin. Item 4 above narrows what we read, but closing the write side needs a change outside OpenMetadata.

Checklist:

• I have read the CONTRIBUTING document.
• My PR title is Fixes <issue-number>: <short explanation>
• My PR is linked to a GitHub issue via Fixes #<issue-number> above.
• I have commented on my code, particularly in hard-to-understand areas.
• I have added tests (unit / integration / Playwright as applicable) and listed them above.
• I have added a test that covers the exact scenario we are fixing.

🤖 Generated with Claude Code

Confidence Score: 4/5

The PR is not yet safe to merge because prefix-scoped deployments still cannot apply the hybrid RRF pipeline or saved ranking weights.

Findings

1.  Inline pipeline is unused ▶

Summary

This PR scopes alias and template discovery to deployment-specific index patterns, makes an indeterminate chunk probe non-fatal, and begins replacing the cluster-global hybrid pipeline with per-query pipeline definitions.

• Uses index-scoped alias requests for OpenSearch and Elasticsearch.
• Restricts template fingerprint lookup to the configured cluster alias.
• Skips chunk staging safely when the live target cannot be determined.
• Makes search-settings saves tolerate named-pipeline update failures.
• Adds an inline RRF pipeline definition, but does not yet attach it to search requests.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Save hybrid search weights] --> B[Attempt named pipeline PUT]
  B -->|Prefix-scoped role rejects PUT| C[Log warning and continue]
  C --> D[Build hybrid search request]
  D -. no caller of inline definition .-> E[Request sent without RRF pipeline]
  E --> F[Saved keyword and semantic weights are not applied]

Loading

_{Reviews (2) · Last reviewed commit: "Fixes #33587: let hybrid ranking ride al..."}

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• Linked issue Search indexing fails with 'Forbidden access' on cluster-alias-scoped search roles #33572 is not added to the Shipping project.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud