Skip to content

Fix flaky TableMetadataReaderCompressionTest concurrency - #19243

Merged
xiangfu0 merged 1 commit into
apache:masterfrom
xiangfu0:xiangfu0/fix-table-metadata-reader-compression-flake
Aug 13, 2026
Merged

Fix flaky TableMetadataReaderCompressionTest concurrency#19243
xiangfu0 merged 1 commit into
apache:masterfrom
xiangfu0:xiangfu0/fix-table-metadata-reader-compression-flake

Conversation

@xiangfu0

Copy link
Copy Markdown
Contributor

Summary

  • Run the preferred mock compression server with two handler threads.
  • Wait for both independent preferred requests to enter their handlers.
  • Shut down the dedicated test executor during cleanup.

Root cause

The embedded HttpServer used its default executor, which serializes request handlers. This test sends final and retryable segment requests to the same preferred endpoint while deliberately blocking the handler. When the final request entered first, the retryable request could reach its half-deadline and be cancelled before its queued handler incremented the request counter. The test then reported expected [2] but found [1] even though the production deadline behavior was correct.

How to reproduce

On an unfixed checkout, repeatedly run the failing method under JDK 25 with test retries disabled:

for attempt in {1..100}; do
  JAVA_HOME=/path/to/jdk-25 ./mvnw -q -pl pinot-controller \
    -Dtest=TableMetadataReaderCompressionTest#testFinalReplicaKeepsOverallDeadlineWhileAnotherSegmentFallsBack \
    -Dunit.test.rerun.count=0 surefire:test || break
done

The race is recorded in #19065 and in GitHub Actions job 94326703474, where the assertion failed after approximately the retryable half-deadline.

Validation

  • Focused JDK 25 build with retries disabled: passed.
  • 20 independent focused Surefire executions: 20/20 passed.
  • Full TableMetadataReaderCompressionTest: 24 tests passed, 0 failures/errors/skips.
  • spotless:apply, checkstyle:check, license:format, and license:check: passed for pinot-controller.
  • git diff --check: passed.

Closes #19065.

@xiangfu0
xiangfu0 marked this pull request as ready for review August 13, 2026 08:20
@xiangfu0
xiangfu0 requested review from Jackie-Jiang and a lite review from Copilot August 13, 2026 08:20
@xiangfu0 xiangfu0 added the flaky-test Tracks a test that intermittently fails label Aug 13, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a known flake in TableMetadataReaderCompressionTest by ensuring the embedded preferred mock HttpServer can execute concurrent request handlers, matching the test’s intended concurrent deadline behavior.

Changes:

  • Run the preferred mock compression server with a dedicated 2-thread executor to avoid serialized handler execution.
  • Update the preferred-handler synchronization to wait for two independent preferred requests to enter the blocking handler.
  • Clean up the new executor during test teardown to avoid thread leakage.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@codecov-commenter

codecov-commenter commented Aug 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 66.94%. Comparing base (cb89c0e) to head (d37e9a6).

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19243      +/-   ##
============================================
- Coverage     66.97%   66.94%   -0.04%     
- Complexity     1417     1423       +6     
============================================
  Files          3453     3453              
  Lines        218858   218858              
  Branches      34787    34787              
============================================
- Hits         146591   146522      -69     
- Misses        60556    60616      +60     
- Partials      11711    11720       +9     
Flag Coverage Δ
integration 100.00% <ø> (+100.00%) ⬆️
integration1 100.00% <ø> (?)
integration2 0.00% <ø> (ø)
java-25 66.94% <ø> (-0.04%) ⬇️
lane-a 100.00% <ø> (+100.00%) ⬆️
lane-b 0.00% <ø> (ø)
temurin 66.94% <ø> (-0.04%) ⬇️
unittests 66.94% <ø> (-0.04%) ⬇️
unittests1 57.70% <ø> (-0.02%) ⬇️
unittests2 39.02% <ø> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xiangfu0
xiangfu0 merged commit 65752d1 into apache:master Aug 13, 2026
14 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

flaky-test Tracks a test that intermittently fails

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Flaky-test] TableMetadataReaderCompressionTest.testFinalReplicaKeepsOverallDeadlineWhileAnotherSegmentFallsBack

4 participants