Document namespace index templates and overlap resolution - #7705
Document namespace index templates and overlap resolution#7705vishaangelova wants to merge 1 commit into
Conversation
Co-authored-by: Cursor <cursoragent@cursor.com>
Elastic Docs AI PR menuCheck the box to run an AI review for this pull request.
Powered by GitHub Agentic Workflows and docs-actions. For more information, reach out to the docs team. |
🔍 Preview links for changed docs |
Elastic Docs Style Checker (Vale)Summary: 1 suggestion found 💡 Suggestions (1): Optional style improvements. Apply when helpful.
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
|
|
||
| # Customize data streams with namespace index templates [data-streams-namespace-custom] | ||
|
|
||
| Namespace index templates are {{fleet}}-managed index templates scoped to one namespace of an installed integration. When you opt a namespace in, {{fleet}} creates one of these templates for each data stream defined by the integration package, and each template references a `<namespace>@custom` component template that you create and manage. |
There was a problem hiding this comment.
Should we put a link to https://www.elastic.co/docs/manage-data/data-store/templates#index-templates on "index templates"?
|
|
||
| The namespaces you opt in for an integration are the same across the whole cluster. The index templates and component templates {{fleet}} creates are also shared across the cluster. | ||
|
|
||
| If your {{kib}} space restricts which namespaces are allowed, you can only enable namespace index templates for namespaces that match those restrictions. |
There was a problem hiding this comment.
Side note: this feature seems pretty hidden. It is not exposed in the UI and the only reference I found is https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/fleet/dev_docs/space_awareness.md. Even the two API endpoints are not surfaced in Kibana APIs reference since they lack an OAS tag. I'm not sure if this was intentional (implementation PR for reference), maybe something to look at.
|
|
||
| # Customize data streams with namespace index templates [data-streams-namespace-custom] | ||
|
|
||
| Namespace index templates are {{fleet}}-managed index templates scoped to one namespace of an installed integration. When you opt a namespace in, {{fleet}} creates one of these templates for each data stream defined by the integration package, and each template references a `<namespace>@custom` component template that you create and manage. |
There was a problem hiding this comment.
Is "integration package" the accepted wording or should it be just "integration"? (I know this has come up before 😅)
| 3. Name the template using the pattern `<namespace>@custom` (for example, `production@custom`). | ||
| 4. Add the index settings, mappings, or aliases you want applied to every data stream in that namespace for opted-in integrations. | ||
| 5. Create the component template. | ||
| 6. Roll over the affected data streams so new backing indices pick up the changes. For example: |
There was a problem hiding this comment.
Nit: this is fine, but how about saying "each" affected data stream to emphasize that it needs to be done for all of them?
|
|
||
| Use this path only when you understand which template should win for each data stream. | ||
|
|
||
| 1. Review every overlapping template's priority against `250`. You can't change the priority of the {{fleet}}-managed namespace index template; it's always `250`. |
There was a problem hiding this comment.
manage_index_templates cluster privilege can change the priority of namespace index templates. These are created with a fixed priority which is the base template priority + 50, i.e.:
- 250 for standard data streams (base 200)
- 200 for
dataset_is_prefixdata streams (base 150)
However an importantly, Fleet will reset the priority on the next package reinstall or upgrade, so it is not advised to change the priority manually unless it's specifically necessary.
I would suggest something along these lines:
Fleet sets the priority of namespace index templates to 250 for most integrations (200 for integrations with dataset_is_prefix: true). While you can change the priority directly using the Elasticsearch Index Template API, Fleet will restore the default value whenever the integration's templates are reinstalled or resynchronized.
|
Thanks so much @vishaangelova this looks great and it should be easy to add #7701 on top of it. I spotted one statement that needs correcting: #7705 (comment) I realise this didn't come up in our discussion, apologies about that. Please let me know if I can clarify anything. My other comments are minor nits. |
2 participants
Summary
This PR documents the Fleet integration namespace index templates feature, which lets you apply shared settings and mappings across all of an integration's data streams in a single namespace, instead of customizing each data stream individually. The feature is GA in Serverless and targets Stack 9.5+.
It also adds a companion page for resolving overlap warnings Fleet surfaces when a user's pre-existing (higher-priority) index templates conflict with the new Fleet-managed namespace index templates.
Implementation PRs:
Resolves #6545 and #7683
Changes
reference/fleet/data-streams-namespace-custom.md(new): How-to for the feature — what namespace index templates are, the<type>-<dataset>@namespace.<namespace>naming and priority (250) scheme, the user-managed<namespace>@customcomponent template, thecomposed_ofprecedence order, UI opt-in (integration Settings tab and policy editor Advanced options), the opt-in APIs (namespace_customization_enabled_forand the_bulk_namespace_customizationendpoint), and how to turn the feature off.reference/fleet/data-streams-template-conflicts.md(new): Troubleshooting page for overlapping index templates — when overlaps arise, how Fleet detects and reports them (preflight_simulate_indexcheck, priority-based outcomes), and three resolution paths (use Fleet-managed templates, keep existing templates, or combine both).reference/fleet/data-streams.md: Adds the namespace-level@customtier to the Edit the Elasticsearch index template section and clarifies the@customprecedence chain (data stream > namespace > package > type).toc.yml.Scope
The ILM policy picker is intentionally out of scope and will be documented separately as part of #7701.
Generative AI disclosure
Tool(s) and model(s) used: Cursor / Claude Opus 4.8