Skip to content

[feat](condition cache) Support condition cache for external table#60897

Open
jacktengg wants to merge 2 commits intoapache:masterfrom
jacktengg:condition-cache-cc
Open

[feat](condition cache) Support condition cache for external table#60897
jacktengg wants to merge 2 commits intoapache:masterfrom
jacktengg:condition-cache-cc

Conversation

@jacktengg
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 27, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@jacktengg
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 28620 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f140c7df2398a491f977da25906a10cce20fee5, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17630	4532	4309	4309
q2	q3	10648	780	520	520
q4	4678	348	260	260
q5	7547	1194	1014	1014
q6	177	173	146	146
q7	766	834	666	666
q8	9287	1479	1350	1350
q9	4867	4715	4653	4653
q10	7155	1891	1627	1627
q11	474	252	239	239
q12	705	565	469	469
q13	17817	4230	3445	3445
q14	229	231	223	223
q15	960	801	798	798
q16	751	726	670	670
q17	698	845	443	443
q18	5980	5338	5173	5173
q19	1201	972	610	610
q20	511	482	389	389
q21	4622	1900	1375	1375
q22	339	282	241	241
Total cold run time: 97042 ms
Total hot run time: 28620 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4444	4366	4359	4359
q2	q3	1759	2169	1726	1726
q4	848	1158	758	758
q5	4049	4298	4330	4298
q6	183	177	142	142
q7	1730	1606	1504	1504
q8	2423	2669	2541	2541
q9	7395	7464	7349	7349
q10	2630	2914	2453	2453
q11	526	438	416	416
q12	505	578	446	446
q13	3928	4528	3739	3739
q14	291	307	282	282
q15	848	809	790	790
q16	731	798	720	720
q17	1167	1508	1367	1367
q18	7086	6795	6584	6584
q19	857	982	912	912
q20	2097	2129	2048	2048
q21	4013	3639	3429	3429
q22	456	456	392	392
Total cold run time: 47966 ms
Total hot run time: 46255 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184197 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9f140c7df2398a491f977da25906a10cce20fee5, data reload: false

query5	5159	638	530	530
query6	338	231	209	209
query7	4233	481	264	264
query8	329	240	227	227
query9	8726	2744	2730	2730
query10	560	388	315	315
query11	16928	17594	17394	17394
query12	202	132	131	131
query13	1430	498	359	359
query14	7049	3234	3018	3018
query14_1	2881	2828	3036	2828
query15	205	200	186	186
query16	1152	558	460	460
query17	2154	720	612	612
query18	2857	513	393	393
query19	207	215	193	193
query20	137	124	126	124
query21	214	137	119	119
query22	5860	5635	5110	5110
query23	17167	16772	16582	16582
query23_1	16696	16629	16626	16626
query24	7071	1587	1198	1198
query24_1	1210	1257	1226	1226
query25	537	451	395	395
query26	1226	261	144	144
query27	2795	473	281	281
query28	4513	1860	1857	1857
query29	787	545	465	465
query30	308	246	205	205
query31	854	708	654	654
query32	81	68	68	68
query33	508	340	275	275
query34	898	903	588	588
query35	618	670	621	621
query36	1092	1101	1020	1020
query37	134	95	83	83
query38	2914	2889	2832	2832
query39	896	867	840	840
query39_1	832	830	834	830
query40	226	155	133	133
query41	62	61	58	58
query42	109	99	103	99
query43	370	381	341	341
query44	
query45	196	191	180	180
query46	877	995	599	599
query47	2113	2143	2029	2029
query48	305	303	225	225
query49	624	463	372	372
query50	677	283	213	213
query51	4152	4049	4077	4049
query52	107	110	97	97
query53	287	333	285	285
query54	299	258	253	253
query55	88	88	90	88
query56	320	306	305	305
query57	1358	1344	1295	1295
query58	303	272	299	272
query59	2553	2770	2606	2606
query60	331	336	318	318
query61	150	145	148	145
query62	611	601	533	533
query63	306	277	274	274
query64	4841	1351	1081	1081
query65	
query66	1389	467	379	379
query67	16235	16471	16453	16453
query68	
query69	401	330	292	292
query70	991	928	959	928
query71	343	310	301	301
query72	3012	2814	2530	2530
query73	563	542	314	314
query74	9939	9886	9749	9749
query75	2824	2750	2454	2454
query76	2302	1030	664	664
query77	356	376	303	303
query78	11167	11319	10690	10690
query79	3020	804	598	598
query80	1763	614	535	535
query81	590	288	243	243
query82	992	150	115	115
query83	340	257	246	246
query84	251	113	99	99
query85	917	473	422	422
query86	519	322	296	296
query87	3105	3061	3088	3061
query88	3507	2640	2632	2632
query89	423	366	337	337
query90	1936	171	167	167
query91	155	146	132	132
query92	79	76	69	69
query93	1425	814	519	519
query94	641	303	299	299
query95	577	388	310	310
query96	630	517	230	230
query97	2463	2531	2411	2411
query98	235	213	209	209
query99	996	970	911	911
Total cold run time: 258686 ms
Total hot run time: 184197 ms

@jacktengg
Copy link
Contributor Author

run p0

@jacktengg
Copy link
Contributor Author

run external

@jacktengg
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 28745 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2b882d226e94fc72abb8bf9704a41541423407c4, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17635	4548	4354	4354
q2	q3	10647	792	521	521
q4	4680	352	257	257
q5	7536	1204	1025	1025
q6	174	177	148	148
q7	783	858	676	676
q8	9308	1448	1307	1307
q9	4863	4727	4712	4712
q10	6817	1876	1622	1622
q11	454	256	246	246
q12	695	563	465	465
q13	17772	4203	3443	3443
q14	228	228	206	206
q15	967	793	783	783
q16	754	716	671	671
q17	710	852	411	411
q18	6017	5397	5263	5263
q19	1113	985	613	613
q20	491	496	387	387
q21	4308	1854	1400	1400
q22	335	279	235	235
Total cold run time: 96287 ms
Total hot run time: 28745 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4437	4362	4361	4361
q2	q3	1769	2177	1732	1732
q4	849	1156	750	750
q5	4018	4330	4316	4316
q6	178	177	141	141
q7	1723	1599	1476	1476
q8	2439	2653	2538	2538
q9	7332	7571	7417	7417
q10	2693	2809	2453	2453
q11	505	442	414	414
q12	503	600	468	468
q13	3932	4426	3702	3702
q14	305	312	290	290
q15	867	812	843	812
q16	733	772	710	710
q17	1167	1526	1329	1329
q18	7220	6885	6749	6749
q19	925	857	887	857
q20	2121	2190	2042	2042
q21	3912	3675	3380	3380
q22	489	429	382	382
Total cold run time: 48117 ms
Total hot run time: 46319 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184276 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2b882d226e94fc72abb8bf9704a41541423407c4, data reload: false

query5	5055	644	524	524
query6	325	220	196	196
query7	4212	474	268	268
query8	342	244	224	224
query9	8737	2723	2705	2705
query10	570	388	344	344
query11	16940	16810	16608	16608
query12	191	131	124	124
query13	1282	452	355	355
query14	6516	3247	2942	2942
query14_1	2858	2827	2845	2827
query15	213	196	185	185
query16	990	457	441	441
query17	1041	689	572	572
query18	2588	425	335	335
query19	200	203	176	176
query20	138	127	126	126
query21	219	144	120	120
query22	5020	5784	5503	5503
query23	17583	17244	17155	17155
query23_1	17034	17001	17002	17001
query24	7378	1659	1230	1230
query24_1	1237	1259	1239	1239
query25	590	492	425	425
query26	1235	262	159	159
query27	2759	475	297	297
query28	4496	1866	1864	1864
query29	833	575	489	489
query30	308	247	208	208
query31	862	725	656	656
query32	81	72	75	72
query33	536	346	297	297
query34	936	900	568	568
query35	653	673	607	607
query36	1117	1163	946	946
query37	151	99	83	83
query38	2919	2881	2880	2880
query39	1035	871	863	863
query39_1	839	828	833	828
query40	234	158	138	138
query41	67	64	63	63
query42	112	105	105	105
query43	376	387	356	356
query44	
query45	200	189	182	182
query46	864	996	603	603
query47	2160	2127	2049	2049
query48	311	316	234	234
query49	645	476	378	378
query50	682	280	225	225
query51	4124	4141	4013	4013
query52	106	108	99	99
query53	305	335	303	303
query54	327	314	262	262
query55	85	85	80	80
query56	313	304	319	304
query57	1376	1329	1272	1272
query58	290	280	271	271
query59	2558	2599	2557	2557
query60	344	341	324	324
query61	150	143	149	143
query62	627	595	551	551
query63	313	269	278	269
query64	4904	1271	984	984
query65	
query66	1408	457	371	371
query67	16378	16366	16244	16244
query68	
query69	400	297	280	280
query70	976	1003	985	985
query71	339	304	299	299
query72	2789	2683	2432	2432
query73	539	540	320	320
query74	9977	9860	9761	9761
query75	2830	2741	2459	2459
query76	2313	1030	668	668
query77	355	388	318	318
query78	11193	11360	10692	10692
query79	1134	795	599	599
query80	1358	644	544	544
query81	553	276	252	252
query82	1014	159	119	119
query83	342	259	238	238
query84	250	119	97	97
query85	909	480	441	441
query86	417	320	278	278
query87	3111	3084	2943	2943
query88	3526	2663	2662	2662
query89	418	376	347	347
query90	1971	172	165	165
query91	169	156	135	135
query92	78	78	73	73
query93	973	817	509	509
query94	639	327	298	298
query95	581	396	315	315
query96	631	515	224	224
query97	2471	2511	2438	2438
query98	231	211	213	211
query99	1002	1033	909	909
Total cold run time: 254353 ms
Total hot run time: 184276 ms

@jacktengg jacktengg force-pushed the condition-cache-cc branch 5 times, most recently from 6781f14 to 15fff1b Compare February 28, 2026 11:00
@jacktengg
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 29158 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 15fff1b61996813f8ea89e9a4a8b70b3bfd7b798, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17612	4540	4303	4303
q2	q3	10648	775	526	526
q4	4681	363	268	268
q5	7544	1215	1016	1016
q6	173	176	148	148
q7	789	876	686	686
q8	9306	1511	1333	1333
q9	4964	4715	4718	4715
q10	6841	1886	1659	1659
q11	455	269	250	250
q12	696	574	466	466
q13	17782	4269	3459	3459
q14	236	241	220	220
q15	936	802	788	788
q16	775	727	680	680
q17	751	868	440	440
q18	6061	5438	5299	5299
q19	1255	987	652	652
q20	512	500	396	396
q21	5002	2001	1581	1581
q22	395	324	273	273
Total cold run time: 97414 ms
Total hot run time: 29158 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4688	4538	4574	4538
q2	q3	1798	2238	1771	1771
q4	869	1210	784	784
q5	4044	4417	4300	4300
q6	189	175	141	141
q7	1775	1647	1505	1505
q8	2517	2897	2555	2555
q9	7482	7351	7321	7321
q10	2597	2789	2398	2398
q11	516	436	411	411
q12	511	585	465	465
q13	4026	4481	3685	3685
q14	287	299	277	277
q15	904	805	793	793
q16	696	758	724	724
q17	1220	1473	1379	1379
q18	7271	6746	6597	6597
q19	922	915	913	913
q20	2191	2127	2024	2024
q21	3994	3504	3394	3394
q22	477	433	367	367
Total cold run time: 48974 ms
Total hot run time: 46342 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183955 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 15fff1b61996813f8ea89e9a4a8b70b3bfd7b798, data reload: false

query5	4560	647	532	532
query6	330	219	215	215
query7	4222	474	273	273
query8	362	255	242	242
query9	8778	2728	2740	2728
query10	499	391	352	352
query11	17065	17602	17287	17287
query12	202	142	130	130
query13	1334	508	381	381
query14	7164	3360	2997	2997
query14_1	2929	2893	2964	2893
query15	247	214	182	182
query16	1015	473	472	472
query17	1119	744	647	647
query18	2709	429	342	342
query19	197	202	176	176
query20	138	125	127	125
query21	214	131	110	110
query22	4734	4849	4916	4849
query23	17253	16723	16566	16566
query23_1	16644	16662	16545	16545
query24	7134	1625	1222	1222
query24_1	1222	1198	1228	1198
query25	539	452	407	407
query26	1239	293	149	149
query27	2753	482	283	283
query28	4522	1855	1871	1855
query29	784	567	464	464
query30	310	244	210	210
query31	883	738	643	643
query32	77	73	69	69
query33	512	348	274	274
query34	905	893	560	560
query35	630	674	599	599
query36	1086	1124	997	997
query37	136	94	82	82
query38	2971	2937	2872	2872
query39	881	874	831	831
query39_1	826	823	828	823
query40	230	156	139	139
query41	63	60	60	60
query42	109	108	105	105
query43	387	387	361	361
query44	
query45	230	189	178	178
query46	871	988	601	601
query47	2105	2147	2033	2033
query48	305	315	243	243
query49	627	468	377	377
query50	686	278	212	212
query51	4083	4107	4100	4100
query52	105	106	98	98
query53	291	339	288	288
query54	293	267	259	259
query55	86	87	77	77
query56	325	303	302	302
query57	1363	1357	1279	1279
query58	288	275	264	264
query59	2617	2718	2595	2595
query60	330	326	333	326
query61	149	146	143	143
query62	622	597	529	529
query63	315	276	281	276
query64	4863	1282	995	995
query65	
query66	1400	455	360	360
query67	16484	16325	16236	16236
query68	
query69	397	305	297	297
query70	958	963	982	963
query71	334	307	301	301
query72	2991	2825	2568	2568
query73	548	555	336	336
query74	9975	9936	9717	9717
query75	2859	2776	2472	2472
query76	2288	1046	692	692
query77	393	426	319	319
query78	11325	11574	10860	10860
query79	1775	784	598	598
query80	1365	626	543	543
query81	572	283	252	252
query82	998	146	113	113
query83	366	261	240	240
query84	256	133	99	99
query85	1050	501	427	427
query86	408	331	329	329
query87	3140	3081	3022	3022
query88	3569	2681	2681	2681
query89	436	372	345	345
query90	1884	170	176	170
query91	166	157	137	137
query92	83	80	67	67
query93	1077	877	514	514
query94	636	333	309	309
query95	589	401	310	310
query96	634	517	229	229
query97	2458	2495	2402	2402
query98	231	221	220	220
query99	1004	1002	922	922
Total cold run time: 254763 ms
Total hot run time: 183955 ms

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 35.69% (121/339) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.63% (19663/37359)
Line Coverage 36.24% (183623/506679)
Region Coverage 32.54% (142483/437833)
Branch Coverage 33.46% (61743/184537)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 64.31% (218/339) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.47% (26146/36581)
Line Coverage 54.24% (273986/505126)
Region Coverage 51.63% (228177/441970)
Branch Coverage 52.84% (97811/185101)

@jacktengg jacktengg force-pushed the condition-cache-cc branch from 15fff1b to aa2b046 Compare March 1, 2026 13:57
@jacktengg jacktengg force-pushed the condition-cache-cc branch from aa2b046 to d1f9381 Compare March 1, 2026 14:45
@jacktengg jacktengg force-pushed the condition-cache-cc branch from da407e2 to a1f386e Compare March 1, 2026 15:24
@jacktengg
Copy link
Contributor Author

run buildall

1 similar comment
@jacktengg
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 28659 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a1f386ed66ee671131b2827eb5e919d580052911, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17629	4515	4276	4276
q2	q3	10647	780	515	515
q4	4673	343	252	252
q5	7559	1189	1027	1027
q6	181	172	145	145
q7	774	825	667	667
q8	9286	1459	1296	1296
q9	4894	4705	4687	4687
q10	6816	1871	1620	1620
q11	439	255	259	255
q12	717	566	481	481
q13	17779	4245	3415	3415
q14	228	230	209	209
q15	961	788	791	788
q16	755	719	669	669
q17	706	883	404	404
q18	5885	5335	5303	5303
q19	1124	983	611	611
q20	500	491	384	384
q21	4492	1857	1413	1413
q22	336	282	242	242
Total cold run time: 96381 ms
Total hot run time: 28659 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4432	4366	4335	4335
q2	q3	1757	2166	1718	1718
q4	849	1144	763	763
q5	4041	4330	4314	4314
q6	170	171	141	141
q7	1704	1563	1487	1487
q8	2411	2665	2511	2511
q9	7226	7762	7367	7367
q10	2733	2854	2414	2414
q11	518	426	412	412
q12	505	575	446	446
q13	3920	4476	3619	3619
q14	318	327	309	309
q15	860	804	847	804
q16	702	749	713	713
q17	1204	1517	1288	1288
q18	7040	6868	6544	6544
q19	885	903	876	876
q20	2063	2153	2049	2049
q21	4010	3756	3334	3334
q22	505	430	374	374
Total cold run time: 47853 ms
Total hot run time: 45818 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184191 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a1f386ed66ee671131b2827eb5e919d580052911, data reload: false

query5	4728	626	495	495
query6	326	214	206	206
query7	4248	482	269	269
query8	351	238	239	238
query9	8767	2739	2727	2727
query10	479	372	337	337
query11	16958	16858	16590	16590
query12	186	128	123	123
query13	1269	450	377	377
query14	6449	3198	2979	2979
query14_1	2813	2784	2771	2771
query15	211	197	180	180
query16	993	472	433	433
query17	1076	718	606	606
query18	2602	448	352	352
query19	210	203	182	182
query20	138	132	132	132
query21	224	146	125	125
query22	4761	6202	5534	5534
query23	17916	17269	17209	17209
query23_1	17058	16950	17081	16950
query24	7284	1608	1240	1240
query24_1	1237	1237	1219	1219
query25	554	484	427	427
query26	1229	266	158	158
query27	2760	461	288	288
query28	4500	1851	1866	1851
query29	777	555	467	467
query30	310	244	207	207
query31	889	716	671	671
query32	78	73	70	70
query33	508	338	276	276
query34	916	912	569	569
query35	641	672	603	603
query36	1093	1138	1010	1010
query37	132	91	80	80
query38	2979	2915	2870	2870
query39	1025	864	839	839
query39_1	829	832	825	825
query40	230	145	132	132
query41	64	57	59	57
query42	104	102	100	100
query43	397	390	349	349
query44	
query45	202	188	184	184
query46	864	972	606	606
query47	2123	2141	2048	2048
query48	329	351	238	238
query49	617	464	382	382
query50	683	278	215	215
query51	4089	4081	4047	4047
query52	106	107	97	97
query53	292	338	275	275
query54	291	261	260	260
query55	100	81	83	81
query56	310	300	300	300
query57	1351	1339	1277	1277
query58	293	278	274	274
query59	2557	2670	2508	2508
query60	322	323	327	323
query61	150	143	143	143
query62	616	599	547	547
query63	324	278	282	278
query64	4848	1273	969	969
query65	
query66	1414	454	356	356
query67	16278	16307	16366	16307
query68	
query69	398	315	287	287
query70	988	951	964	951
query71	330	314	298	298
query72	2771	2601	2372	2372
query73	533	545	315	315
query74	9985	9868	9720	9720
query75	2872	2734	2467	2467
query76	2296	1042	702	702
query77	366	371	295	295
query78	11197	11369	10695	10695
query79	3018	816	611	611
query80	1751	636	517	517
query81	598	275	244	244
query82	995	147	114	114
query83	327	267	240	240
query84	253	124	96	96
query85	879	459	438	438
query86	497	312	296	296
query87	3087	3094	3003	3003
query88	3541	2648	2641	2641
query89	418	365	343	343
query90	2253	176	163	163
query91	162	158	134	134
query92	86	85	70	70
query93	2044	840	521	521
query94	644	312	280	280
query95	588	395	313	313
query96	643	511	231	231
query97	2478	2518	2433	2433
query98	238	218	214	214
query99	1015	980	912	912
Total cold run time: 257240 ms
Total hot run time: 184191 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants