Skip to content

Fix ES metrics reads locating indices with superDatasetDayStep instead of dayStep - #14139

Merged
wu-sheng merged 1 commit into
apache:masterfrom
Dev-next-gen:fix/es-metrics-index-day-step
Oct 15, 2026
Merged

wu-sheng merged 1 commit into
apache:masterfrom
Dev-next-gen:fix/es-metrics-index-day-step

Conversation

@Dev-next-gen

Copy link
Copy Markdown
Contributor

Fix the Elasticsearch storage reading metrics from the wrong daily indices when dayStep and superDatasetDayStep differ

  • Add a unit test to verify that the fix works.
  • Explain briefly why the bug exists and how to fix it.

Metrics are written to indices rolled by dayStep (TimeSeriesUtils.writeIndexName), and readMetricsValues reads them back the same way through queryIndexName. Four other metrics reads locate their indices with TimeRangeIndexNameGenerator, which calls superDatasetIndexNames and rolls by superDatasetDayStep:

  • AggregationQueryEsDAO.sortMetrics (top N)
  • MetricsQueryEsDAO.readLabeledMetricsValuesWithoutEntity
  • MetricsQueryEsDAO.listEntityIdsInRange
  • both TagAutoCompleteQueryDAO queries (TagAutocompleteData is a metrics model, not a super dataset)

With the default superDatasetDayStep: -1 both steps are equal and nothing shows. Once they differ, these reads search indices that do not hold the data. The search is sent with ignore_unavailable=true, so the missing indices are skipped silently and the query returns nothing or only part of the range.

Concrete case, the configuration TimeSeriesUtilsTest already uses: dayStep: 3, superDatasetDayStep: 1. The minutes of 2020-08-09 are written to <metric>-20200807, but a top N over 2020-08-09 10:10 to 10:20 searches only <metric>-20200809.

The fix adds TimeSeriesUtils.metricsIndexNames, the same range walk as superDatasetIndexNames but rolled by dayStep, and these four reads use it. The super dataset readers (traces, logs, Zipkin, OTLP spans, AI agent and GenAI evaluation records) keep TimeRangeIndexNameGenerator. In TagAutoCompleteQueryDAO, the time bounds become final so the lambda can capture them.

AggregationQueryEsDAOTest runs sortMetrics against a mocked client with the steps above and checks the indices it searches. Before the fix it fails with ["service_resp_time-20200809"] where ["service_resp_time-20200807"] is expected. After it, the test passes, as do the 34 tests of storage-elasticsearch-plugin, and checkstyle is clean.

  • If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.
  • Update the CHANGES log.

Found by a defect-hunting pipeline I build and run (Dev-next-gen), using Claude Code with Anthropic's Claude Opus 5.5.

Metrics are written to indices rolled by dayStep, but the top N query,
the labeled metrics read without an entity, the entity listing and the
tag autocomplete query built their index names with
superDatasetDayStep. When the two steps differ, these reads search
indices that do not hold the data, and Elasticsearch ignores the
missing ones, so they return nothing or only part of the range.
@wu-sheng wu-sheng added this to the 11.1.0 milestone Oct 15, 2026
@wu-sheng wu-sheng added the bug Something isn't working and you are sure it's a bug! label Oct 15, 2026

@wu-sheng wu-sheng left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merging for now as it is valid bug fixing. We will refactor codes to provide unified APIs to reduce the confusion for developers.

@Dev-next-gen

Copy link
Copy Markdown
Contributor Author

Thanks for the review and for taking the fix ahead of the API unification. I checked the five red checks on 673b912 since they are what keeps Required red; none of them comes from the changed ES read paths.

Alarm ES and Alarm ES Sharding also failed on #14135 a few hours earlier, which does not touch ES storage (Alarm ES, Alarm ES Sharding), and on runs 38340517699 and 38181306104 as well; here Alarm ES passed 13 cases and failed one after 30 retries on queryAlarms returning msgs: []. Agent Python failed on instance list: the consumer's container hostname was 845886322e50, which the expected-file template read as a number (float64(8.45886322e+58) in the diff). Rover with Istio Process 1.28.0 stopped at Maven resolution, No versions available for io.grpc:grpc-core:jar:[1.43.2] within specified range, before any test ran. In the same run Storage ES 9.3.0, Storage ES 8.18.8, Storage ES Sharding 8.18.8, the three Storage OpenSearch jobs and Baseline-driven Alarm ES are green.

I tried gh run rerun 38667069305 --failed but it needs admin rights, so if a re-run of the failed jobs is the simplest way to get Required green, it would have to come from your side.

Found by a defect-hunting pipeline I build and run (Dev-next-gen), using Claude Code with Anthropic's Claude Opus 5.5.

@wu-sheng
wu-sheng merged commit cf1c2be into apache:master Oct 15, 2026
476 of 482 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working and you are sure it's a bug!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants