Skip to content

Query fuzz tests: remove the #7550 duplicate-series error workaround once Prometheus >= v3.15.0 is vendored #7862

Description

@sandy2008

AI Tool Usage Notice
If you used an AI tool to help draft this issue,
please make sure you have reviewed and validated all content before submitting.
You are responsible for the accuracy and quality of everything in this request.
Low-quality or unreviewed AI-generated submissions may be closed without further investigation.
See our Generative AI Contribution Policy for details.

Is your feature request related to a problem? Please describe.
This tracks the follow-up agreed in #7550. That PR added a tactical workaround to integration/query_fuzz_test.go for the flake in #7546. The PromQL engines emitted found duplicate series for the match group …: [X, Y] with the two series in non-deterministic order, so Cortex and the reference engine could report the same error as [X, Y] vs [Y, X]. The workaround has two parts:

  • duplicateSeriesRE and canonicalizeDuplicateSeriesErr canonicalize that one message before comparing. sameErrorClass uses them at four sites: TestDisableChunkTrimmingFuzz, TestExpandedPostingsCacheFuzz, TestLazyMatchersFuzz and runQueryFuzzTestCases.
  • TestSameErrorClass tests the helper.

As agreed in the #7550 review, it should be removed "once both upstream fixes are vendored", so the fuzz tests don't permanently encode upstream error-message formats. That is currently blocked.

Status as of 2026-09-25:

Cortex can't simply bump Prometheus. Newer Prometheus releases no longer ship github.com/prometheus/prometheus/tsdb/errors; it is absent from both v3.14.0 and v3.15.0-rc.1. That package is imported by 6 Cortex files and by the vendored Thanos (pkg/block/writer.go). Building master against v0.315.0-rc.1 fails on the Thanos import. Thanos main is still on Prometheus v3.9.1, and its upgrade PR, thanos-io/thanos#8774 (to 3.11), is a draft with conflicts.

Describe the solution you'd like
Once Cortex vendors Prometheus ≥ v3.15.0, which requires a Thanos version that supports it:

  1. Bump images.Prometheus in integration/e2e/images/images.go to ≥ v3.15.0.
  2. Delete duplicateSeriesRE, canonicalizeDuplicateSeriesErr and TestSameErrorClass. Go back to a strict error comparison at the four call sites, as before fix(integration): compare query fuzz errors by class, not exact string #7550.
  3. Keep a tolerance for TestBackwardCompatibilityQueryFuzz until the latest Cortex release includes the fix. That test compares against getLatestReleaseImage(), whose engine still emits the old order. Alternatively, do the removal in two steps: everything else first, and the backward-compatibility case one release later.

Both upstream fixes put the smaller rendered label set first, so strict comparison will also hold between the Prometheus and Thanos engines.

Describe alternatives you've considered

Additional context

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions