Repository navigation
Fix ES metrics reads locating indices with superDatasetDayStep instead of dayStep - #14139
Conversation
Metrics are written to indices rolled by dayStep, but the top N query, the labeled metrics read without an entity, the entity listing and the tag autocomplete query built their index names with superDatasetDayStep. When the two steps differ, these reads search indices that do not hold the data, and Elasticsearch ignores the missing ones, so they return nothing or only part of the range.
wu-sheng
left a comment
There was a problem hiding this comment.
Merging for now as it is valid bug fixing. We will refactor codes to provide unified APIs to reduce the confusion for developers.
|
Thanks for the review and for taking the fix ahead of the API unification. I checked the five red checks on 673b912 since they are what keeps
I tried Found by a defect-hunting pipeline I build and run (Dev-next-gen), using Claude Code with Anthropic's Claude Opus 5.5. |
Fix the Elasticsearch storage reading metrics from the wrong daily indices when
dayStepandsuperDatasetDayStepdifferMetrics are written to indices rolled by
dayStep(TimeSeriesUtils.writeIndexName), andreadMetricsValuesreads them back the same way throughqueryIndexName. Four other metrics reads locate their indices withTimeRangeIndexNameGenerator, which callssuperDatasetIndexNamesand rolls bysuperDatasetDayStep:AggregationQueryEsDAO.sortMetrics(top N)MetricsQueryEsDAO.readLabeledMetricsValuesWithoutEntityMetricsQueryEsDAO.listEntityIdsInRangeTagAutoCompleteQueryDAOqueries (TagAutocompleteDatais a metrics model, not a super dataset)With the default
superDatasetDayStep: -1both steps are equal and nothing shows. Once they differ, these reads search indices that do not hold the data. The search is sent withignore_unavailable=true, so the missing indices are skipped silently and the query returns nothing or only part of the range.Concrete case, the configuration
TimeSeriesUtilsTestalready uses:dayStep: 3,superDatasetDayStep: 1. The minutes of 2020-08-09 are written to<metric>-20200807, but a top N over 2020-08-09 10:10 to 10:20 searches only<metric>-20200809.The fix adds
TimeSeriesUtils.metricsIndexNames, the same range walk assuperDatasetIndexNamesbut rolled bydayStep, and these four reads use it. The super dataset readers (traces, logs, Zipkin, OTLP spans, AI agent and GenAI evaluation records) keepTimeRangeIndexNameGenerator. InTagAutoCompleteQueryDAO, the time bounds become final so the lambda can capture them.AggregationQueryEsDAOTestrunssortMetricsagainst a mocked client with the steps above and checks the indices it searches. Before the fix it fails with["service_resp_time-20200809"]where["service_resp_time-20200807"]is expected. After it, the test passes, as do the 34 tests ofstorage-elasticsearch-plugin, and checkstyle is clean.CHANGESlog.Found by a defect-hunting pipeline I build and run (Dev-next-gen), using Claude Code with Anthropic's Claude Opus 5.5.