Skip to content

[improvement](topn) Refresh TopN runtime filter page ranges - #70020

Open
jacktengg wants to merge 2 commits into
apache:masterfrom
jacktengg:armp-perf-page-prune
Open

jacktengg wants to merge 2 commits into
apache:masterfrom
jacktengg:armp-perf-page-prune

Conversation

@jacktengg

@jacktengg jacktengg commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Problem Summary:

Segment readers keep row ranges computed before a TopN runtime bound arrives or tightens. Scan-key normalization can also remove a shared runtime predicate on a leading storage key. Readers then miss later bound updates.

Publish versioned predicate snapshots and use page Zone Maps to refresh unread row ranges between batches. Apply the first bound immediately and throttle later refreshes by the number of raw rows read. Preserve scan progress, bitmap holes, and read direction. Refresh the bitmap cursor, cached cardinality, and filter counters together. Keep shared runtime predicates during static scan-key normalization while still removing static predicates covered by scan keys.

In the existing ascending TopN check with 1,048,576 rows and three sort keys, ScanRows fell from 1,048,576 to 16,384 and TotalPagesNum fell from 372 to 6. These figures compare filtering disabled and enabled on the same patched build. The query results were identical.

Release note

TopN scans can skip unread pages when runtime bounds arrive or tighten. Runtime predicates remain active after scan-key normalization.

Check List (For Author)

  • Test: Manual checks of commit boundaries, dependencies, generated-result preservation, whitespace, clang-format v16, and header hygiene. Include BE unit tests and two TopN regression suites. Builds and tests were not rerun for this commit split.
  • Behavior changed: Yes. TopN scans can read fewer rows and pages. This commit retains inclusive runtime bounds.
  • Does this need documentation: No

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69776

Problem Summary:

Segment readers keep row ranges computed before a TopN runtime bound arrives
or tightens. Scan-key normalization can also remove a shared runtime predicate
on a leading storage key. Readers then miss later bound updates.

Publish versioned predicate snapshots and use page Zone Maps to refresh unread
row ranges between batches. Apply the first bound immediately and throttle
later refreshes by the number of raw rows read. Preserve scan progress, bitmap
holes, and read direction. Refresh the bitmap cursor, cached cardinality, and
filter counters together. Keep shared runtime predicates during static scan-key
normalization while still removing static predicates covered by scan keys.

In the existing ascending TopN check with 1,048,576 rows and three sort keys,
ScanRows fell from 1,048,576 to 16,384 and TotalPagesNum fell from 372 to 6.
These figures compare filtering disabled and enabled on the same patched build.
The query results were identical.

### Release note

TopN scans can skip unread pages when runtime bounds arrive or tighten.
Runtime predicates remain active after scan-key normalization.

### Check List (For Author)

- Test: Manual checks of commit boundaries, dependencies, generated-result
  preservation, whitespace, clang-format v16, and header hygiene. Include BE
  unit tests and two TopN regression suites. Builds and tests were not rerun
  for this commit split.
- Behavior changed: Yes. TopN scans can read fewer rows and pages. This commit
  retains inclusive runtime bounds.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@jacktengg

Copy link
Copy Markdown
Contributor Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#70020
commit: 1b8d5fd63c7547c2964ccf45cf6ce32a5540b659
base: 156dbe71dc63ed6e919c131d33279264b73187f6
reviewed_at: 2026-11-03T12:12+08:00
reviewer: jacktengg
model: gpt-6-astra
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • Source review covered all 10 changed files. Pure-text BE build-hygiene checks passed; builds, unit/regression tests, and benchmarks were not executed in this review.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

@jacktengg

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28745 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1b8d5fd63c7547c2964ccf45cf6ce32a5540b659, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17588	5163	5041	5041
q2	2129	319	291	291
q3	10187	1412	824	824
q4	4682	463	334	334
q5	7442	801	537	537
q6	180	182	152	152
q7	742	789	609	609
q8	9253	1352	1024	1024
q9	5532	4196	4179	4179
q10	6838	1289	1041	1041
q11	428	245	226	226
q12	642	400	280	280
q13	18111	3602	3055	3055
q14	283	281	250	250
q15	q16	762	720	686	686
q17	1215	773	585	585
q18	7075	6206	6139	6139
q19	1262	966	596	596
q20	398	339	228	228
q21	5936	2379	2460	2379
q22	376	337	289	289
Total cold run time: 101061 ms
Total hot run time: 28745 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5989	5931	6158	5931
q2	732	620	609	609
q3	6019	5815	5349	5349
q4	1983	2161	1564	1564
q5	5697	5901	5655	5655
q6	239	185	144	144
q7	2249	2086	1795	1795
q8	2991	2636	2660	2636
q9	8002	7544	7561	7544
q10	3653	3588	3166	3166
q11	575	401	373	373
q12	664	681	513	513
q13	3311	3678	3064	3064
q14	284	281	260	260
q15	q16	673	686	620	620
q17	7730	7057	6872	6872
q18	12957	12188	12838	12188
q19	880	783	829	783
q20	2125	2141	1924	1924
q21	5564	4580	4704	4580
q22	495	452	391	391
Total cold run time: 72812 ms
Total hot run time: 65961 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151820 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1b8d5fd63c7547c2964ccf45cf6ce32a5540b659, data reload: false

query5	4327	589	451	451
query6	429	204	199	199
query7	4797	439	230	230
query8	315	166	150	150
query9	8757	3156	3168	3156
query10	486	314	261	261
query11	5988	3893	3566	3566
query12	155	95	90	90
query13	1261	462	347	347
query14	6147	3925	3656	3656
query14_1	3411	3483	3448	3448
query15	218	204	196	196
query16	947	438	421	421
query17	874	653	533	533
query18	2430	426	310	310
query19	204	163	139	139
query20	85	87	86	86
query21	211	128	109	109
query22	12932	13075	12843	12843
query23	13514	13023	12474	12474
query23_1	12579	12460	12688	12460
query24	7400	822	479	479
query24_1	489	485	494	485
query25	536	423	366	366
query26	1273	253	140	140
query27	2776	444	275	275
query28	4621	2042	1997	1997
query29	1574	603	461	461
query30	313	223	189	189
query31	1009	947	792	792
query32	152	90	95	90
query33	535	304	254	254
query34	914	862	499	499
query35	732	743	648	648
query36	849	812	751	751
query37	136	94	90	90
query38	1796	1716	1645	1645
query39	801	788	733	733
query39_1	747	735	735	735
query40	229	116	102	102
query41	72	68	72	68
query42	93	86	84	84
query43	345	359	311	311
query44	1303	711	709	709
query45	182	178	167	167
query46	844	968	565	565
query47	2969	3033	2714	2714
query48	301	309	215	215
query49	577	397	291	291
query50	671	286	217	217
query51	10837	10693	10739	10693
query52	80	79	75	75
query53	189	207	151	151
query54	245	188	177	177
query55	71	64	62	62
query56	235	218	209	209
query57	1679	1624	1598	1598
query58	287	265	269	265
query59	2196	2220	2029	2029
query60	291	249	218	218
query61	179	151	144	144
query62	388	325	284	284
query63	195	156	158	156
query64	2754	919	776	776
query65	3370	3288	3246	3246
query66	1779	418	296	296
query67	20240	20086	20207	20086
query68	3005	1017	610	610
query69	376	285	260	260
query70	872	779	758	758
query71	298	228	228	228
query72	2770	2586	2280	2280
query73	517	529	290	290
query74	4615	4501	4349	4349
query75	2340	2295	1930	1930
query76	2308	1019	589	589
query77	337	365	270	270
query78	10015	9809	9214	9214
query79	1007	928	543	543
query80	1363	439	375	375
query81	599	328	273	273
query82	799	137	113	113
query83	293	201	183	183
query84	311	118	108	108
query85	1018	422	405	405
query86	403	239	233	233
query87	2010	1940	1799	1799
query88	3804	2786	2760	2760
query89	337	295	248	248
query90	1859	179	165	165
query91	170	156	128	128
query92	93	89	92	89
query93	1031	1019	583	583
query94	642	306	274	274
query95	577	320	301	301
query96	694	501	240	240
query97	2445	2472	2317	2317
query98	163	151	151	151
query99	774	778	667	667
Total cold run time: 236994 ms
Total hot run time: 151820 ms

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 58.33% (7/12) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 67.29% (32102/47705)
Line Coverage 52.10% (337452/647761)
Region Coverage 47.63% (274315/575976)
Branch Coverage 49.47% (129317/261387)

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.07 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1b8d5fd63c7547c2964ccf45cf6ce32a5540b659, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.04	0.04
query3	0.25	0.08	0.09
query4	1.62	0.11	0.11
query5	0.24	0.24	0.24
query6	1.16	0.68	0.68
query7	0.04	0.01	0.01
query8	0.06	0.03	0.03
query9	0.39	0.35	0.32
query10	0.58	0.58	0.61
query11	0.20	0.15	0.14
query12	0.18	0.15	0.15
query13	0.52	0.49	0.51
query14	1.03	1.01	1.00
query15	0.64	0.60	0.59
query16	0.34	0.34	0.35
query17	1.16	1.14	1.16
query18	0.20	0.18	0.19
query19	2.15	1.97	2.04
query20	0.02	0.01	0.02
query21	15.51	0.19	0.14
query22	4.94	0.06	0.06
query23	16.28	0.34	0.12
query24	1.08	0.26	0.16
query25	0.10	0.06	0.04
query26	0.74	0.22	0.16
query27	0.03	0.03	0.03
query28	3.73	0.89	0.34
query29	12.50	4.06	3.27
query30	0.28	0.13	0.10
query31	2.79	0.56	0.33
query32	3.23	0.60	0.50
query33	3.25	3.26	3.19
query34	15.30	3.99	3.38
query35	3.35	3.36	3.35
query36	0.55	0.43	0.43
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.14
query41	0.09	0.04	0.03
query42	0.05	0.03	0.03
query43	0.05	0.03	0.03
Total cold run time: 95.1 s
Total hot run time: 24.07 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 58.33% (7/12) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.68% (35898/46212)
Line Coverage 63.02% (403692/640612)
Region Coverage 59.58% (341557/573264)
Branch Coverage 60.71% (158102/260429)

@jacktengg jacktengg changed the title [improvement](be) Refresh TopN runtime filter page ranges [improvement](topn) Refresh TopN runtime filter page ranges Nov 3, 2026
HappenLee
HappenLee previously approved these changes Nov 3, 2026

@HappenLee HappenLee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the versioned predicate snapshots, unread-range intersection, and bitmap cursor/cardinality updates. Follow up with base/head benchmarks for frequently tightening bounds with low page-pruning selectivity.

### What problem does this PR solve?

Issue Number: None

Related PR: apache#70020

Problem Summary: The SharedPredicate constructor taking another SharedPredicate
and a column ID has no callers in production code or tests. RuntimePredicate
creates instances with a column ID and name, while clone() returns the existing
shared instance. Remove the unused overload without changing runtime behavior.

### Release note

None

### Check List (For Author)

- Test: Manual test: searched all tracked references and verified the factory
  construction and clone paths; clang-format 16 and check-format passed for the
  modified header; BE build-hygiene and git diff --check passed. No build or unit
  tests run for this unused-code deletion. clang-tidy was attempted but could not
  complete because the existing analysis environment cannot locate stddef.h.
- Behavior changed: No
- Does this need documentation: No
@jacktengg

Copy link
Copy Markdown
Contributor Author

run buildall

@jacktengg

Copy link
Copy Markdown
Contributor Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#70020
commit: 0e20c230d359fab1102ca1491d004643f72ecfb9
base: 156dbe71dc63ed6e919c131d33279264b73187f6
reviewed_at: 2026-11-03T21:51+08:00
reviewer: jacktengg
model: gpt-6-astra
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • Static review only; builds, unit/regression tests and benchmarks were not run locally.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

@jacktengg

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28651 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0e20c230d359fab1102ca1491d004643f72ecfb9, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17582	5144	5027	5027
q2	2103	318	298	298
q3	10197	1407	823	823
q4	4683	465	342	342
q5	7429	795	528	528
q6	178	186	151	151
q7	772	802	591	591
q8	9258	1350	1087	1087
q9	5434	4197	4233	4197
q10	6830	1298	1014	1014
q11	436	242	233	233
q12	642	404	274	274
q13	18108	5267	3069	3069
q14	271	264	241	241
q15	q16	740	732	655	655
q17	1208	753	561	561
q18	7040	6205	6142	6142
q19	1262	953	603	603
q20	379	343	228	228
q21	5654	2320	2422	2320
q22	385	339	267	267
Total cold run time: 100591 ms
Total hot run time: 28651 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5981	5917	6156	5917
q2	757	609	558	558
q3	5670	5918	5319	5319
q4	2042	2158	1596	1596
q5	5720	5827	5640	5640
q6	242	191	147	147
q7	2250	1992	1760	1760
q8	3016	2639	2636	2636
q9	7848	7524	7520	7520
q10	3626	3582	3148	3148
q11	580	414	394	394
q12	656	694	507	507
q13	3269	3588	3024	3024
q14	291	293	263	263
q15	q16	662	681	617	617
q17	7737	7057	6834	6834
q18	12964	12167	12866	12167
q19	914	780	780	780
q20	2134	2133	1909	1909
q21	5577	4554	4693	4554
q22	492	479	412	412
Total cold run time: 72428 ms
Total hot run time: 65702 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151211 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0e20c230d359fab1102ca1491d004643f72ecfb9, data reload: false

query5	4299	590	447	447
query6	425	210	192	192
query7	4803	442	226	226
query8	315	160	149	149
query9	8752	3164	3152	3152
query10	486	306	249	249
query11	5938	3871	3585	3585
query12	150	90	83	83
query13	1239	464	344	344
query14	6141	3896	3636	3636
query14_1	3418	3390	3422	3390
query15	205	199	179	179
query16	946	396	410	396
query17	885	656	559	559
query18	2409	418	313	313
query19	200	190	134	134
query20	85	86	82	82
query21	209	133	121	121
query22	12909	12987	12768	12768
query23	13378	12864	12450	12450
query23_1	12635	12596	12574	12574
query24	7359	824	484	484
query24_1	485	487	505	487
query25	530	430	363	363
query26	1254	260	138	138
query27	2761	439	273	273
query28	4691	2017	2046	2017
query29	1558	604	452	452
query30	301	226	185	185
query31	1003	926	810	810
query32	142	89	94	89
query33	526	304	245	245
query34	991	849	504	504
query35	734	763	646	646
query36	836	818	730	730
query37	121	95	83	83
query38	1766	1713	1691	1691
query39	790	792	754	754
query39_1	740	721	745	721
query40	235	122	98	98
query41	90	85	68	68
query42	88	86	86	86
query43	342	371	312	312
query44	1311	693	714	693
query45	183	180	161	161
query46	840	956	579	579
query47	3020	2883	2901	2883
query48	318	329	229	229
query49	596	412	289	289
query50	676	275	211	211
query51	10560	10756	10500	10500
query52	76	76	69	69
query53	182	209	149	149
query54	233	229	173	173
query55	73	63	62	62
query56	225	224	210	210
query57	1641	1586	1561	1561
query58	283	263	250	250
query59	2136	2207	2013	2013
query60	289	243	222	222
query61	149	150	144	144
query62	382	335	295	295
query63	187	156	164	156
query64	2757	916	787	787
query65	3309	3257	3265	3257
query66	1766	423	289	289
query67	20489	20299	19859	19859
query68	3286	994	608	608
query69	374	279	240	240
query70	855	788	759	759
query71	304	228	202	202
query72	2823	2593	2335	2335
query73	536	524	295	295
query74	4600	4535	4356	4356
query75	2334	2308	1907	1907
query76	2383	1023	588	588
query77	341	366	272	272
query78	9764	9762	9146	9146
query79	992	865	521	521
query80	1218	436	392	392
query81	598	314	272	272
query82	571	133	108	108
query83	306	197	183	183
query84	324	123	102	102
query85	856	430	370	370
query86	403	235	229	229
query87	2000	1952	1814	1814
query88	3820	2744	2736	2736
query89	331	280	242	242
query90	1919	168	161	161
query91	164	150	123	123
query92	97	91	90	90
query93	1084	996	574	574
query94	640	291	259	259
query95	587	386	303	303
query96	669	484	235	235
query97	2474	2466	2369	2369
query98	158	157	145	145
query99	767	764	660	660
Total cold run time: 236114 ms
Total hot run time: 151211 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 23.97 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0e20c230d359fab1102ca1491d004643f72ecfb9, data reload: false

query1	0.00	0.01	0.00
query2	0.09	0.05	0.05
query3	0.26	0.09	0.09
query4	1.60	0.08	0.11
query5	0.23	0.24	0.25
query6	1.18	0.68	0.67
query7	0.03	0.01	0.01
query8	0.05	0.03	0.03
query9	0.40	0.35	0.35
query10	0.57	0.60	0.60
query11	0.19	0.14	0.14
query12	0.18	0.14	0.16
query13	0.49	0.48	0.49
query14	1.04	1.01	1.01
query15	0.61	0.59	0.60
query16	0.31	0.34	0.35
query17	1.15	1.11	1.11
query18	0.19	0.18	0.18
query19	2.05	2.00	2.03
query20	0.02	0.01	0.01
query21	15.51	0.19	0.13
query22	4.88	0.06	0.05
query23	16.28	0.34	0.12
query24	1.07	0.30	0.17
query25	0.08	0.05	0.05
query26	0.77	0.22	0.15
query27	0.03	0.03	0.04
query28	3.86	0.74	0.38
query29	12.51	4.04	3.21
query30	0.27	0.12	0.09
query31	2.77	0.54	0.32
query32	3.23	0.58	0.51
query33	3.11	3.19	3.18
query34	15.36	3.97	3.35
query35	3.30	3.32	3.34
query36	0.55	0.47	0.44
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.13
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 94.74 s
Total hot run time: 23.97 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (7/7) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.76% (35947/46226)
Line Coverage 63.16% (404814/640941)
Region Coverage 59.86% (343304/573547)
Branch Coverage 60.93% (158833/260688)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (7/7) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.76% (35947/46226)
Line Coverage 63.16% (404814/640941)
Region Coverage 59.82% (343123/573547)
Branch Coverage 60.93% (158826/260688)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants