Repository navigation
perf(query)!: rank drill-down traces from per-minute span exemplars - #504
Merged
Merged
Conversation
Drill-down ranks every retained trace in its window, so it times out once the server holds data. Propose exactly maintained per-bucket rank lists in the read cache, and add the filebench scale benchmark that measures the current cost.
Rank drill-down traces by their best in-scope span and cache bounded per-batch, scope and minute exemplar lists with no cross-batch state. Update the read-cache rules in AGENTS.md to match.
Service-map drill-down and the recent-trace lookup ranked every retained in-window trace candidate, and each read-cache pass merged new parts into that retained per-trace index, so both grew with retained traces. A trace now scores by its best in-scope, in-window span: an error first, then the longest span, then trace_id and namespace. The append-only read_trace_exemplars table keeps, per batch, namespace, service and minute of span start, the best span of the top 21 traces (full rows) and the spans fewer than 21 other traces outrank from either side (suffix and prefix rows). A window reads full rows of covered minutes, the partial start and end minutes' suffix and prefix rows, and the uncached files' raw spans, then takes each trace's best score. A window inside a single minute reads the overlapping files' raw spans. Rows are written with the batch markers, retention deletes them by batch, and compaction ranks the union of its inputs' rows again in the marker transfer. The candidate and part tables, the retired-trace repair and the partial-trace raw path are removed; the read cache version is now 20.
Spread the scale fixture over PITOT_SCALE_HOURS, rank the newest unaligned 24 hours, and report read-cache pass times, exemplar rows, and snapshot binding apart from execution.
Retire an error span from the same namespace and service as the trace's kept span and assert its rank and error flag drop. Rank windows across negative minute boundaries against brute force and check each row's floored minute. Build the trace_exemplars source only for statements that reference it.
vishr
marked this pull request as ready for review
November 18, 2026 20:06
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Drill-down ranked every retained trace in its window, and the read-cache candidate MERGE behind it joined every retained trace on each pass. Both grew with retained data, so a server that had ingested for a while timed out on
/api/queries/exemplars. This replaces that state with bounded, append-only span exemplar lists per batch, scope and minute, as accepted in proposal 0012.Ranking change. Drill-down and
observability.Service.Tracenow rank a trace by its best in-scope, in-window span: error first, then longer duration, thentrace_id,namespace. Error-first selection and every displayed field are unchanged. Which traces fill the 20 slots can differ from the old extent ranking (max_end - min_start), which the UI never showed. The generic exemplar path already ranked by span duration.Changes
read_trace_exemplarsholds skyline rows (in_full,in_suffix,in_prefix) per (batch, namespace, service, minute). They are written in the completed-batch transaction with the markers, deleted by batch on retention, and re-ranked from the inputs' rows on compaction. An uncovered compaction rebuilds from Parquet.CompletedTraceQuerykeeps its arguments and output. A window reads full rows of covered minutes, suffix and prefix rows of its edge minutes, and the raw spans of uncached batches throughUNION ALL. A window inside one minute reads footer-pruned Parquet.read_trace_candidates,read_trace_parts,trace_tail,traceCandidateSource, the retired-trace repair and the partial-trace raw path, andread_batches.replacement_id.readCacheVersion19 → 20. The private read tables rebuild once on upgrade; no telemetry is lost.Verification
TestCompletedTraceQueryMatchesBruteForcecompares the query with brute force over raw spans. It covers six scopes, K of 1 and 21, ties, the same trace ID in two namespaces, sub-minute and unaligned windows, late publication, an uncached tail, covered and uncovered compaction across generations, and retention. Deliberate mutations make it fail. A second oracle covers pre-epoch minutes.filebench, 24h window, median): 11–13 ms at 1.6M traces and 12–14 ms at 16M traces over 240h of retention. With all 16M traces inside the window: 24–26 ms. The old query took 831 ms (single-service) and 2,426 ms (two-service) at 16M.just test,just vetandjust check-staticpass.A Hetzner pitot-bench run with a full-day prefill follows the release.
Related: #480, labstack/pitot-bench#14.