Fix GenesByGenericSpliceSites performance (6.5min -> 0.6s) - #232
Open
jbrestel wants to merge 1 commit into
Open
Conversation
Restrict the sample aggregation to the two chosen samples up front so the indexes on protocol_app_node_id apply. Previously all 48 samples were aggregated -- twice, since the stats subquery repeated the same join -- and filtered afterwards. Compute the predominant-site max in a single window pass. The old self-join on sum_cpm = max_cpm compared double precision values produced by two separate aggregations, which are not guaranteed bit-identical and could drop a gene's max row. Pass organismAbbrev from the injected question (the injector already calls setOrganismAbbrevFromDatasetName) so the partitioned webready _p tables prune to one organism instead of scanning 800+ partitions, and read webready.TranscriptAttributes_p rather than the unpartitioned view. Materialize the pairing CTE. Inlined, Postgres flattens it into a cross product joined by filter -- 35M rows generated and discarded on tbruTREU927 -- because a CTE has no statistics for it to estimate from. Timed on all six sample pairs across every organism with a differential question: worst case 1.0s, previously unusable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The differential splice-site search (
GenesByDifferentialSpliceSites<dataset>, onequestion per dataset, all sharing
GeneId.GenesByGenericSpliceSites) did not completeon T. brucei. Four independent problems; the last one accounts for almost all the time.
1. The sample filter never reached the scans
loc_countsaggregated the fullsplicesitegenes ⋈ splicesitefeaturejoin for all 48protocol_app_node_ids, andstatsrepeated the identical join and aggregate — Postgresdoes not share a CTE it inlines. The chosen samples were applied afterwards as a filter on
the CTE scan. Restricting
loc_countsto the two samples up front lets the existingssf_revix1index apply.2. Two float aggregations compared for equality
WHERE this.sum_cpm = stats.max_cpmcompareddouble precisionsums from two separateaggregation passes. Not guaranteed bit-identical, so a gene's max row could silently drop.
Replaced with a single window pass (
sum_cpm = max(sum_cpm) OVER w), where both valuescome from the same pass.
3. No partition key on the
webready_ptableswebready.SpliceSiteTranscript_pis partitioned onorg_abbrevand the query supplied nopredicate, so it scanned all 831 partitions, twice (1,662 partition scans, a 3,369-line
plan). The question is per-dataset and therefore per-organism, so the abbrev is known at
model-build time:
spliceSitesDifferentialQuestionnow declaresprop=organismAbbrevandpasses it as a hidden paramRef, the same arrangement
GenesByGoTermCLDatasetuses. Noinjector change needed —
SpliceSites.javaalready callssetOrganismAbbrevFromDatasetName().Also switched
apidbtuning.TranscriptAttributestowebready.TranscriptAttributes_p.4. The pairing CTE was inlined into a cross product
This was the actual cost. A CTE referenced once is inlined, so Postgres dissolved the
one/twopairing and instead joined each sample's transcripts independently, then pairedevery row with every row:
35.1M rows generated and discarded — 377 s of the 390 s. It chose this because a CTE has no
statistics, so it estimated
rows=1per side and believed the cross product was one row.AS MATERIALIZEDonpairs(andgene_loc_stats) fences it.Verification
Timed against
genomicsdb_071n, all six sample pairs — every organism with a differentialquestion. Results verified identical to the pre-change query where it completed at all.
tbruTREU927/44316+44307 went from 390 s to 0.60 s. Also run end-to-end in the app on adev instance, which exercises WDK's
CREATE TABLE … ASresult-cache wrapper.Notes for the reviewer
144k-cost plan ran 1 second. Please judge changes here by timing, not
EXPLAINcost.g.protocol_app_node_id IN (...)toloc_countsis logically safe(0 disagreeing rows of 3.7M) but takes the query from 1 s to 500 s — it switches
splicesitegenesoff its index-only scan and collapses the row estimates. Deliberatelynot included.
wdkQuery -showQueryis not currently possible:GeneVQ.SpliceSiteExperimentreturns
external_database_release_idasinternal, while every per-dataset overridereturns
ed.name, which is what the dependent sample vocab needs. Dormant (the overridealways wins) and left alone here to keep the diff focused.
🤖 Generated with Claude Code