Skip to content

fix: tolerate incomplete query traces in ShardAwarenessTest - #1000

Merged
dkropachev merged 1 commit into
scylladb:scylla-3.xfrom
nikagra:fix/shardawareness-trace-flake
Aug 13, 2026
Merged

fix: tolerate incomplete query traces in ShardAwarenessTest#1000
dkropachev merged 1 commit into
scylladb:scylla-3.xfrom
nikagra:fix/shardawareness-trace-flake

Conversation

@nikagra

@nikagra nikagra commented Aug 13, 2026

Copy link
Copy Markdown

correctShardInTracingTest fails on ~27% of java-driver-matrix-test builds (master apache#2083, apache#2085, apache#2091; 2026.3 #4), always with:

No 'querying locally' trace event was observed for the query

QueryTrace.doFetchTrace() retries only until the coordinator's session row reports a duration, then freezes whatever the concurrently-issued events query returned — its own comment notes the trace "may not contain the log of replicas". So getEvents() can return an empty list, which the test read as "the coordinator never queried locally".

Retry the traced read up to 3 times, binding a fresh statement per attempt (reusing one would pin the coordinator via the paging optimization), and fail only when no attempt observes a local read. Re-executing is required: once fetched, a trace's events are cached. A wrong shard still fails immediately, without retrying.

Verified against 3-node Scylla 2026.1: 10/10 runs pass, one of which hit the real fault and recovered —

Attempt 1/3 for pk=100002: trace fa0a9170-… carried no 'querying locally'
    event (0 events); retrying with a fresh trace

pk=100002 is the shard-1 key that fails in CI. Sabotaging the matcher so nothing can match still yields BUILD FAILURE after 3 attempts.

🤖 Generated with Claude Code

QueryTrace.doFetchTrace() retries only until the coordinator's session row
reports a duration; the events fetched alongside it can still be empty. The
test read that as "the coordinator never queried locally" and failed, which
reddened the driver matrix on roughly a quarter of runs.

Retry the traced read up to 3 times, binding a fresh statement each attempt,
and fail only when no attempt observes a local read. A wrong shard still
fails immediately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Aug 13, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: QUIET

Plan: Pro Plus

Run ID: 9e9a30e2-6f9f-43bc-beec-be50f8a5816f

📥 Commits

Reviewing files that changed from the base of the PR and between 855dc8c and 428cc25.

📒 Files selected for processing (1)
  • driver-core/src/test/java/com/datastax/driver/core/ShardAwarenessTest.java
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

  • scylladb/github-automation (auto-detected)
  • scylladb/scylladb (auto-detected)

📝 Walkthrough

Walkthrough

ShardAwarenessTest now retries traced shard verification up to three times. Each attempt uses a fresh bound statement, validates the returned row, and checks for a data-local read on the expected shard. The test ignores unrelated coordinator events, normalizes service-level thread suffixes, logs incomplete traces, and fails explicitly when no valid local-read event appears.

Possibly related issues

Possibly related PRs

Mergeability Score: ⚪ Minimal · up to 428cc

This PR makes the shard-awareness test retry incomplete traces while preserving immediate failure for wrong-shard results. No actionable merge-blocking risk remains beyond normal checks and review.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the fix for incomplete query traces in ShardAwarenessTest.
Description check ✅ Passed The description directly explains the intermittent failure, retry strategy, fresh statements, and validation behavior.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Comment @coderabbitai help to get the list of available commands.

@dkropachev
dkropachev merged commit a75a7ca into scylladb:scylla-3.x Aug 13, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants