You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
AI Tool Usage Notice
If you used an AI tool to help draft this issue,
please make sure you have reviewed and validated all content before submitting.
You are responsible for the accuracy and quality of everything in this request.
Low-quality or unreviewed AI-generated submissions may be closed without further investigation.
See our Generative AI Contribution Policy for details.
Is your feature request related to a problem? Please describe.
This tracks the follow-up agreed in #7550. That PR added a tactical workaround to integration/query_fuzz_test.go for the flake in #7546. The PromQL engines emitted found duplicate series for the match group …: [X, Y] with the two series in non-deterministic order, so Cortex and the reference engine could report the same error as [X, Y] vs [Y, X]. The workaround has two parts:
duplicateSeriesRE and canonicalizeDuplicateSeriesErr canonicalize that one message before comparing. sameErrorClass uses them at four sites: TestDisableChunkTrimmingFuzz, TestExpandedPostingsCacheFuzz, TestLazyMatchersFuzz and runQueryFuzzTestCases.
TestSameErrorClass tests the helper.
As agreed in the #7550 review, it should be removed "once both upstream fixes are vendored", so the fuzz tests don't permanently encode upstream error-message formats. That is currently blocked.
Cortex: vendors Prometheus v0.309.1 (v3.9.1). images.Prometheus, the reference server used by TestPrometheusCompatibilityQueryFuzz, TestNativeHistogramFuzz, TestExperimentalPromQLFuncsWithPrometheus and TestStoreGatewayLazyExpandedPostingsSeriesFuzzWithPrometheus, is quay.io/prometheus/prometheus:v3.9.1.
Cortex can't simply bump Prometheus. Newer Prometheus releases no longer ship github.com/prometheus/prometheus/tsdb/errors; it is absent from both v3.14.0 and v3.15.0-rc.1. That package is imported by 6 Cortex files and by the vendored Thanos (pkg/block/writer.go). Building master against v0.315.0-rc.1 fails on the Thanos import. Thanos main is still on Prometheus v3.9.1, and its upgrade PR, thanos-io/thanos#8774 (to 3.11), is a draft with conflicts.
Describe the solution you'd like
Once Cortex vendors Prometheus ≥ v3.15.0, which requires a Thanos version that supports it:
Bump images.Prometheus in integration/e2e/images/images.go to ≥ v3.15.0.
Keep a tolerance for TestBackwardCompatibilityQueryFuzz until the latest Cortex release includes the fix. That test compares against getLatestReleaseImage(), whose engine still emits the old order. Alternatively, do the removal in two steps: everything else first, and the backward-compatibility case one release later.
Both upstream fixes put the smaller rendered label set first, so strict comparison will also hold between the Prometheus and Thanos engines.
AI Tool Usage Notice
If you used an AI tool to help draft this issue,
please make sure you have reviewed and validated all content before submitting.
You are responsible for the accuracy and quality of everything in this request.
Low-quality or unreviewed AI-generated submissions may be closed without further investigation.
See our Generative AI Contribution Policy for details.
Is your feature request related to a problem? Please describe.
This tracks the follow-up agreed in #7550. That PR added a tactical workaround to
integration/query_fuzz_test.gofor the flake in #7546. The PromQL engines emittedfound duplicate series for the match group …: [X, Y]with the two series in non-deterministic order, so Cortex and the reference engine could report the same error as[X, Y]vs[Y, X]. The workaround has two parts:duplicateSeriesREandcanonicalizeDuplicateSeriesErrcanonicalize that one message before comparing.sameErrorClassuses them at four sites:TestDisableChunkTrimmingFuzz,TestExpandedPostingsCacheFuzz,TestLazyMatchersFuzzandrunQueryFuzzTestCases.TestSameErrorClasstests the helper.As agreed in the #7550 review, it should be removed "once both upstream fixes are vendored", so the fuzz tests don't permanently encode upstream error-message formats. That is currently blocked.
Status as of 2026-09-25:
1b359e941bdeincludes it.mainon 2026-08-17. The first release that contains it is v3.15.0, currently atv3.15.0-rc.1. It is not in v3.14.0 or v3.13.3.v0.309.1(v3.9.1).images.Prometheus, the reference server used byTestPrometheusCompatibilityQueryFuzz,TestNativeHistogramFuzz,TestExperimentalPromQLFuncsWithPrometheusandTestStoreGatewayLazyExpandedPostingsSeriesFuzzWithPrometheus, isquay.io/prometheus/prometheus:v3.9.1.Cortex can't simply bump Prometheus. Newer Prometheus releases no longer ship
github.com/prometheus/prometheus/tsdb/errors; it is absent from both v3.14.0 and v3.15.0-rc.1. That package is imported by 6 Cortex files and by the vendored Thanos (pkg/block/writer.go). Building master againstv0.315.0-rc.1fails on the Thanos import. Thanosmainis still on Prometheus v3.9.1, and its upgrade PR, thanos-io/thanos#8774 (to 3.11), is a draft with conflicts.Describe the solution you'd like
Once Cortex vendors Prometheus ≥ v3.15.0, which requires a Thanos version that supports it:
images.Prometheusinintegration/e2e/images/images.goto ≥ v3.15.0.duplicateSeriesRE,canonicalizeDuplicateSeriesErrandTestSameErrorClass. Go back to a strict error comparison at the four call sites, as before fix(integration): compare query fuzz errors by class, not exact string #7550.TestBackwardCompatibilityQueryFuzzuntil the latest Cortex release includes the fix. That test compares againstgetLatestReleaseImage(), whose engine still emits the old order. Alternatively, do the removal in two steps: everything else first, and the backward-compatibility case one release later.Both upstream fixes put the smaller rendered label set first, so strict comparison will also hold between the Prometheus and Thanos engines.
Describe alternatives you've considered
tsdb/errors.Additional context
github.com/prometheus/prometheusto ≥v0.315.0, plus Thanos.