Skip to content

perf(query)!: rank drill-down traces from per-minute span exemplars - #504

Merged
vishr merged 6 commits into
mainfrom
docs/proposal-0012-bounded-trace-ranking
Nov 18, 2026
Merged

vishr merged 6 commits into
mainfrom
docs/proposal-0012-bounded-trace-ranking

Conversation

@vishr

@vishr vishr commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Summary

Drill-down ranked every retained trace in its window, and the read-cache candidate MERGE behind it joined every retained trace on each pass. Both grew with retained data, so a server that had ingested for a while timed out on /api/queries/exemplars. This replaces that state with bounded, append-only span exemplar lists per batch, scope and minute, as accepted in proposal 0012.

Ranking change. Drill-down and observability.Service.Trace now rank a trace by its best in-scope, in-window span: error first, then longer duration, then trace_id, namespace. Error-first selection and every displayed field are unchanged. Which traces fill the 20 slots can differ from the old extent ranking (max_end - min_start), which the UI never showed. The generic exemplar path already ranked by span duration.

Changes

  • read_trace_exemplars holds skyline rows (in_full, in_suffix, in_prefix) per (batch, namespace, service, minute). They are written in the completed-batch transaction with the markers, deleted by batch on retention, and re-ranked from the inputs' rows on compaction. An uncovered compaction rebuilds from Parquet.
  • CompletedTraceQuery keeps its arguments and output. A window reads full rows of covered minutes, suffix and prefix rows of its edge minutes, and the raw spans of uncached batches through UNION ALL. A window inside one minute reads footer-pruned Parquet.
  • Removed: read_trace_candidates, read_trace_parts, trace_tail, traceCandidateSource, the retired-trace repair and the partial-trace raw path, and read_batches.replacement_id.
  • readCacheVersion 19 → 20. The private read tables rebuild once on upgrade; no telemetry is lost.
  • The exemplar source is built only for statements that reference it.
  • AGENTS.md read-cache rules and proposal 0012 updated.

Verification

  • TestCompletedTraceQueryMatchesBruteForce compares the query with brute force over raw spans. It covers six scopes, K of 1 and 21, ties, the same trace ID in two namespaces, sub-minute and unaligned windows, late publication, an uncached tail, covered and uncovered compaction across generations, and retention. Deliberate mutations make it fail. A second oracle covers pre-epoch minutes.
  • Scale (filebench, 24h window, median): 11–13 ms at 1.6M traces and 12–14 ms at 16M traces over 240h of retention. With all 16M traces inside the window: 24–26 ms. The old query took 831 ms (single-service) and 2,426 ms (two-service) at 16M.
  • Refresh pass cost is flat from 0.2M to 8M retained exemplar rows (about 10 ms per pass).
  • just test, just vet and just check-static pass.

A Hetzner pitot-bench run with a full-day prefill follows the release.

Related: #480, labstack/pitot-bench#14.

Drill-down ranks every retained trace in its window, so it times out once
the server holds data. Propose exactly maintained per-bucket rank lists in
the read cache, and add the filebench scale benchmark that measures the
current cost.
Rank drill-down traces by their best in-scope span and cache bounded
per-batch, scope and minute exemplar lists with no cross-batch state.
Update the read-cache rules in AGENTS.md to match.
Service-map drill-down and the recent-trace lookup ranked every retained
in-window trace candidate, and each read-cache pass merged new parts into
that retained per-trace index, so both grew with retained traces.

A trace now scores by its best in-scope, in-window span: an error first,
then the longest span, then trace_id and namespace. The append-only
read_trace_exemplars table keeps, per batch, namespace, service and
minute of span start, the best span of the top 21 traces (full rows) and
the spans fewer than 21 other traces outrank from either side (suffix
and prefix rows). A window reads full rows of covered minutes, the
partial start and end minutes' suffix and prefix rows, and the uncached
files' raw spans, then takes each trace's best score. A window inside a
single minute reads the overlapping files' raw spans.

Rows are written with the batch markers, retention deletes them by
batch, and compaction ranks the union of its inputs' rows again in the
marker transfer. The candidate and part tables, the retired-trace repair
and the partial-trace raw path are removed; the read cache version is
now 20.
Spread the scale fixture over PITOT_SCALE_HOURS, rank the newest
unaligned 24 hours, and report read-cache pass times, exemplar rows, and
snapshot binding apart from execution.
Retire an error span from the same namespace and service as the trace's
kept span and assert its rank and error flag drop. Rank windows across
negative minute boundaries against brute force and check each row's
floored minute. Build the trace_exemplars source only for statements
that reference it.
@vishr vishr changed the title docs(proposals): 0012 bounded trace ranking for drill-down perf(query)!: rank drill-down traces from per-minute span exemplars Nov 18, 2026
@vishr
vishr marked this pull request as ready for review November 18, 2026 20:06
@vishr
vishr added this pull request to the merge queue Nov 18, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 18, 2026
@vishr
vishr added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit bc1288d Nov 18, 2026
32 checks passed
@vishr
vishr deleted the docs/proposal-0012-bounded-trace-ranking branch November 18, 2026 22:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant