Skip to content

[fix](be) Own values in single-value aggregate states - #68210

Open
Mryange wants to merge 1 commit into
apache:masterfrom
Mryange:refactor-single-value-state
Open

Mryange wants to merge 1 commit into
apache:masterfrom
Mryange:refactor-single-value-state

Conversation

@Mryange

@Mryange Mryange commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Single-value aggregate state implementations were duplicated across aggregate functions, and complex values were not handled consistently. In particular, window functions such as first_value, last_value, nth_value, lead, and lag retained a raw input column pointer and row offset. The input column can be cleared while the window state is still alive, leaving the state with an invalid reference.

Root cause: the shared state abstraction did not own complex values, so aggregate functions either maintained separate implementations or referenced input columns directly.

This change introduces reusable fixed-width, string, and owned-column single-value states. The column-backed state materializes exactly one selected row and is reused by complex min, max, min_by, max_by, and window value functions. Unsupported complex comparison types use non-incremental window evaluation. Existing fixed-width and string state serialization formats are preserved.

Tests cover value ownership after source-column reclamation, comparison and serialization, and verify that retained column allocation does not scale with the number of source rows.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#67439

Problem Summary: Single-value aggregate states duplicated storage logic, while first_value, last_value, nth_value, lead, and lag retained raw input column references that could outlive reclaimed input columns. Introduce reusable fixed-width, string, and owned-column state implementations, reuse them across min/max, min_by/max_by, any, and window value functions, and keep complex window types on non-incremental evaluation when equality comparison is unsupported.

### Release note

Fix single-value window functions retaining references to reclaimed input columns.

### Check List (For Author)

- Test: Unit Test
    - ./run-be-ut.sh --run --filter=SingleValueDataStringTest.*:SingleValueDataColumnTest.* -j 48 (18 tests passed under ASAN)
- Behavior changed: Yes. Window states now own selected values instead of retaining raw input column references.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Mryange

Mryange commented Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 27815 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 30c7406198fe6cc3cc0e95f6461901c404b1061d, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17713	3978	3879	3879
q2	2289	349	303	303
q3	10028	1456	782	782
q4	4692	475	347	347
q5	7463	809	537	537
q6	190	177	136	136
q7	733	788	596	596
q8	9305	1620	1525	1525
q9	5383	4174	4137	4137
q10	6715	1618	1351	1351
q11	434	289	250	250
q12	627	423	314	314
q13	18041	2616	1981	1981
q14	266	254	237	237
q15	q16	735	721	670	670
q17	1758	1094	1077	1077
q18	6476	5587	5556	5556
q19	1165	1181	1003	1003
q20	477	388	282	282
q21	5720	2956	2555	2555
q22	454	359	297	297
Total cold run time: 100664 ms
Total hot run time: 27815 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4555	4458	4419	4419
q2	775	605	589	589
q3	4786	5297	4674	4674
q4	2255	2345	1460	1460
q5	4551	4417	4444	4417
q6	227	200	132	132
q7	1894	1695	1490	1490
q8	2326	2039	2035	2035
q9	7297	7056	6853	6853
q10	4221	4175	3799	3799
q11	509	415	356	356
q12	703	704	517	517
q13	2277	2601	2000	2000
q14	262	268	250	250
q15	q16	651	688	597	597
q17	7288	6690	6637	6637
q18	11837	11068	11715	11068
q19	1091	977	987	977
q20	2241	2178	1911	1911
q21	5362	4469	4608	4469
q22	496	467	424	424
Total cold run time: 65604 ms
Total hot run time: 59074 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 153951 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 30c7406198fe6cc3cc0e95f6461901c404b1061d, data reload: false

query5	4317	594	473	473
query6	426	182	163	163
query7	4833	528	317	317
query8	338	202	189	189
query9	8779	3938	3930	3930
query10	481	322	252	252
query11	5835	2220	1997	1997
query12	162	97	98	97
query13	1241	536	412	412
query14	6514	4651	4268	4268
query14_1	4058	4240	4032	4032
query15	201	210	177	177
query16	990	504	430	430
query17	906	688	558	558
query18	2419	455	337	337
query19	200	181	144	144
query20	104	103	102	102
query21	217	134	119	119
query22	13103	13000	12830	12830
query23	15469	14563	13969	13969
query23_1	14043	14149	14140	14140
query24	7527	1694	1246	1246
query24_1	1279	1241	1236	1236
query25	543	454	342	342
query26	1263	306	162	162
query27	2712	562	329	329
query28	4592	1951	1943	1943
query29	1042	570	455	455
query30	319	235	200	200
query31	892	761	635	635
query32	142	93	96	93
query33	492	307	234	234
query34	1213	1131	635	635
query35	725	767	643	643
query36	802	806	744	744
query37	161	108	92	92
query38	1812	1756	1697	1697
query39	687	689	677	677
query39_1	657	654	644	644
query40	210	119	104	104
query41	66	64	62	62
query42	94	93	89	89
query43	397	370	339	339
query44	1405	699	700	699
query45	183	181	170	170
query46	1056	1185	695	695
query47	1489	1522	1386	1386
query48	391	392	290	290
query49	597	416	281	281
query50	950	362	253	253
query51	10672	10579	10359	10359
query52	87	89	75	75
query53	239	252	178	178
query54	266	211	195	195
query55	76	80	72	72
query56	239	241	220	220
query57	1525	1467	1413	1413
query58	233	213	211	211
query59	1955	2058	1817	1817
query60	278	238	225	225
query61	144	146	132	132
query62	402	317	268	268
query63	213	171	176	171
query64	2849	1082	844	844
query65	4004	3966	3915	3915
query66	1813	410	312	312
query67	20197	19975	19746	19746
query68	3203	1502	969	969
query69	414	294	270	270
query70	959	919	891	891
query71	306	233	216	216
query72	2985	2494	2176	2176
query73	796	772	464	464
query74	4626	4622	4306	4306
query75	2295	2286	1945	1945
query76	2338	1138	709	709
query77	364	395	296	296
query78	9046	9078	8506	8506
query79	1207	1208	750	750
query80	506	476	380	380
query81	453	284	238	238
query82	264	164	131	131
query83	271	268	245	245
query84	298	137	111	111
query85	779	444	384	384
query86	299	232	233	232
query87	2005	1973	1830	1830
query88	3687	2774	2739	2739
query89	325	291	252	252
query90	2088	183	180	180
query91	165	166	124	124
query92	102	90	90	90
query93	1407	1444	890	890
query94	520	339	294	294
query95	659	370	340	340
query96	1007	785	345	345
query97	2459	2434	2316	2316
query98	199	191	184	184
query99	746	708	599	599
Total cold run time: 240539 ms
Total hot run time: 153951 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 23.81 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 30c7406198fe6cc3cc0e95f6461901c404b1061d, data reload: false

query1	0.01	0.01	0.00
query2	0.10	0.05	0.05
query3	0.26	0.14	0.13
query4	1.61	0.14	0.14
query5	0.24	0.21	0.22
query6	1.16	0.95	0.94
query7	0.04	0.00	0.00
query8	0.05	0.04	0.03
query9	0.38	0.33	0.34
query10	0.54	0.55	0.59
query11	0.20	0.14	0.14
query12	0.18	0.15	0.14
query13	0.47	0.47	0.47
query14	0.97	0.93	0.95
query15	0.59	0.60	0.58
query16	0.30	0.32	0.32
query17	1.06	1.07	1.03
query18	0.21	0.20	0.20
query19	1.98	1.94	1.94
query20	0.02	0.01	0.02
query21	15.43	0.21	0.13
query22	4.94	0.05	0.06
query23	16.15	0.31	0.11
query24	2.96	0.42	0.33
query25	0.10	0.04	0.04
query26	0.73	0.20	0.14
query27	0.04	0.03	0.04
query28	3.54	0.77	0.34
query29	12.56	4.00	3.16
query30	0.28	0.16	0.15
query31	2.77	0.58	0.31
query32	3.22	0.59	0.48
query33	3.12	3.31	3.12
query34	15.81	3.92	3.28
query35	3.22	3.21	3.24
query36	0.55	0.43	0.45
query37	0.09	0.07	0.07
query38	0.05	0.03	0.05
query39	0.04	0.03	0.03
query40	0.18	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 96.32 s
Total hot run time: 23.81 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 95.21% (378/397) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.17% (34289/45019)
Line Coverage 61.12% (385289/630404)
Region Coverage 57.58% (324758/563984)
Branch Coverage 58.31% (147768/253398)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants