fix(dataInsight): preserve service filter when rebuilding terms aggregation

[ulixius9]

Summary

Fixes the bug where Data Insights live system charts return counts for the wrong service. When a chart is requested with a serviceName filter (e.g. the chartDataStream websocket on a service page), the data pushed back belongs to other services.

Closes #28715

Fixes: #28531

Root cause

DataInsightSystemChartRepository.listChartData applies the service filter via includeXAxisFiled = serviceName.toLowerCase(), which the aggregators turn into a terms aggregation on service.name.keyword with .include(regexp(serviceName)).

But every live chart's metric carries a formula, so sub-aggregations are always populated. The aggregator then rebuilds the terms aggregation to attach those sub-aggregations, copying only field + size and silently dropping .include/.exclude:

// before
metricAggregations.put(metricName,
    Aggregation.of(a -> a.terms(t -> t.field(fieldName).size(size))  // include/exclude lost
        .aggregations(subAggregations)));

So the query aggregates over all services (top-100 buckets), returning other services' data. Affects assets_with_description_live, assets_with_pii_live, assets_with_tier_live, assets_with_owner_live, healthy_data_assets, total_data_assets_live, pipeline_status_live.

Fix

Hoist finalIncludeTerms/finalExcludeTerms to loop-body scope and reapply them when rebuilding the terms aggregation, in both ElasticSearchLineChartAggregator and OpenSearchLineChartAggregator.

Tests

Added ElasticSearchLineChartAggregatorTest and OpenSearchLineChartAggregatorTest (4 cases each):

• include filter survives the sub-aggregation rebuild (the bug)
• exclude filter survives the rebuild
• no filter ⇒ terms aggregation stays unfiltered
• include filter survives for grouped charts (total_data_assets_live shape)

The tests build the chart, serialize the resulting SearchRequest to JSON, and assert the service filter is present on the service.name.keyword terms aggregation. They fail without the fix (expected: <myservice> but was: <>).

Verification

mvn -pl openmetadata-service test -Dtest='ElasticSearchLineChartAggregatorTest,OpenSearchLineChartAggregatorTest'
# Tests run: 8, Failures: 0, Errors: 0

mvn spotless:apply is clean.

🤖 Generated with Claude Code

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[github-actions]

🟡 Playwright Results — all passed (8 flaky)

✅ 4272 passed · ❌ 0 failed · 🟡 8 flaky · ⏭️ 88 skipped

Shard	Passed	Failed	Flaky	Skipped
✅ Shard 1	301	0	0	4
🟡 Shard 2	802	0	2	9
🟡 Shard 3	804	0	1	8
🟡 Shard 4	853	0	2	12
✅ Shard 5	721	0	0	47
🟡 Shard 6	791	0	3	8

🟡 8 flaky test(s) (passed on retry)

• Features/DataQuality/DataQuality.spec.ts › TestCase filters (shard 2, 1 retry)
• Features/Glossary/GlossaryWorkflow.spec.ts › should start term as Draft when glossary has reviewers (shard 2, 1 retry)
• Features/KnowledgeCenterList.spec.ts › Knowledge Center List - Test upvote and downvote buttons (shard 3, 1 retry)
• Pages/CustomProperties.spec.ts › Date Time (shard 4, 1 retry)
• Pages/CustomProperties.spec.ts › Should verify property name is visible for apiCollection in right panel (shard 4, 1 retry)
• Pages/Glossary.spec.ts › Column dropdown drag-and-drop functionality for Glossary Terms table (shard 6, 1 retry)
• Pages/Lineage/LineageFilters.spec.ts › Verify lineage service type filter selection (shard 6, 1 retry)
• Pages/Lineage/LineageFilters.spec.ts › Verify lineage schema filter selection (shard 6, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[github-actions]

✅ PR checks passed

The linked issue has a description and all required Shipping project fields set. Thanks!

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[github-actions]

Failed to cherry-pick changes to the 1.13 branch.
Please cherry-pick the changes manually.
You can find more details here.

[gitar-bot]

Code Review ✅ Approved

Restores the missing service filter in Data Insight live chart aggregators by correctly reapplying include/exclude terms during the sub-aggregation rebuild. Verified with new test cases covering both ElasticSearch and OpenSearch implementations.

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}