Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](Nereids): add project for joinreorder #34993

Merged
merged 8 commits into from
May 22, 2024
Merged

[fix](Nereids): add project for joinreorder #34993

merged 8 commits into from
May 22, 2024

Conversation

jackwener
Copy link
Member

Proposed changes

  • Add project for all join reorder rule to confirm column is pruned

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@jackwener
Copy link
Member Author

run buildall

@jackwener
Copy link
Member Author

run buildall

@jackwener
Copy link
Member Author

run buildall

@jackwener jackwener force-pushed the fix_column_3 branch 2 times, most recently from dfc36ba to a2a941a Compare May 20, 2024 12:48
@jackwener
Copy link
Member Author

run buildall

@jackwener jackwener force-pushed the fix_column_3 branch 3 times, most recently from 0dbd279 to 5d995a1 Compare May 20, 2024 13:57
@jackwener
Copy link
Member Author

run buildall

@jackwener
Copy link
Member Author

run buildall

@jackwener
Copy link
Member Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

morrySnow
morrySnow reviewed May 21, 2024
View reviewed changes
...e-core/src/main/java/org/apache/doris/nereids/processor/post/ColumnPruningPostProcessor.java Outdated Show resolved Hide resolved
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/MergeProjectPostProcessor.java Show resolved Hide resolved
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessor.java Outdated Show resolved Hide resolved
...re/src/main/java/org/apache/doris/nereids/rules/exploration/TransposeAggSemiJoinProject.java Outdated Show resolved Hide resolved
regression-test/data/nereids_hint_tpcds_p0/shape/query1.out Show resolved Hide resolved
regression-test/data/nereids_rules_p0/transposeJoin/transposeSemiJoinAgg.out Show resolved Hide resolved
regression-test/data/nereids_tpcds_shape_sf1000_p0/bs_downgrade_shape/query13.out Show resolved Hide resolved
regression-test/suites/query_p0/join/test_join.groovy Show resolved Hide resolved
@jackwener
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40170 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f04014e696834d4afdf647edefb3b32af4060470, data reload: false

------ Round 1 ----------------------------------
q1	18137	4495	4366	4366
q2	2927	195	190	190
q3	11046	1303	1303	1303
q4	10632	791	788	788
q5	7529	2691	2703	2691
q6	222	133	132	132
q7	956	610	613	610
q8	9376	2062	2053	2053
q9	7022	6490	6472	6472
q10	8882	3710	3725	3710
q11	474	241	228	228
q12	456	230	216	216
q13	17769	2975	2986	2975
q14	248	212	234	212
q15	513	472	464	464
q16	483	377	368	368
q17	952	678	729	678
q18	8061	7524	7574	7524
q19	5785	1506	1527	1506
q20	649	315	302	302
q21	4883	3177	3108	3108
q22	349	282	274	274
Total cold run time: 117351 ms
Total hot run time: 40170 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4381	4281	4297	4281
q2	366	265	256	256
q3	2984	2735	2737	2735
q4	1910	1598	1615	1598
q5	5257	5275	5279	5275
q6	210	125	126	125
q7	2145	1779	1699	1699
q8	3235	3290	3270	3270
q9	8332	8389	8351	8351
q10	3930	3700	3730	3700
q11	589	482	501	482
q12	759	594	614	594
q13	17224	2960	2972	2960
q14	278	261	268	261
q15	508	476	464	464
q16	459	435	416	416
q17	1778	1496	1498	1496
q18	7512	7596	7404	7404
q19	1651	1575	1545	1545
q20	1948	1797	1759	1759
q21	4964	4788	4836	4788
q22	574	473	504	473
Total cold run time: 70994 ms
Total hot run time: 53932 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172497 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f04014e696834d4afdf647edefb3b32af4060470, data reload: false

query1	937	381	370	370
query2	6463	2317	2308	2308
query3	6654	219	218	218
query4	19353	17538	17288	17288
query5	4169	415	425	415
query6	253	164	151	151
query7	4591	301	289	289
query8	242	185	179	179
query9	8577	2387	2385	2385
query10	466	276	278	276
query11	10733	10196	10300	10196
query12	147	92	86	86
query13	1660	365	359	359
query14	10044	7729	6797	6797
query15	248	167	179	167
query16	8172	270	263	263
query17	1833	550	549	549
query18	2113	285	280	280
query19	210	169	171	169
query20	93	87	92	87
query21	204	128	128	128
query22	4237	4082	3921	3921
query23	34930	33103	32988	32988
query24	12106	2797	2842	2797
query25	509	404	362	362
query26	1138	165	159	159
query27	2852	327	327	327
query28	7256	2028	2028	2028
query29	863	609	594	594
query30	311	169	182	169
query31	955	768	755	755
query32	95	54	55	54
query33	765	285	265	265
query34	1008	468	477	468
query35	719	601	595	595
query36	1074	919	911	911
query37	195	68	70	68
query38	2900	2750	2763	2750
query39	869	795	816	795
query40	267	128	127	127
query41	46	43	44	43
query42	106	98	97	97
query43	587	561	534	534
query44	1291	727	737	727
query45	180	161	162	161
query46	1075	723	731	723
query47	1860	1742	1783	1742
query48	378	303	297	297
query49	1197	380	383	380
query50	769	399	391	391
query51	6850	6852	6783	6783
query52	109	90	90	90
query53	353	283	279	279
query54	848	428	413	413
query55	73	72	75	72
query56	253	241	243	241
query57	1152	1041	1010	1010
query58	244	208	224	208
query59	3333	3302	3173	3173
query60	276	282	248	248
query61	89	120	89	89
query62	636	454	455	454
query63	312	282	276	276
query64	9656	2191	1727	1727
query65	3641	3122	3111	3111
query66	1241	343	329	329
query67	15372	15236	15002	15002
query68	4848	529	584	529
query69	465	262	269	262
query70	1176	1090	1133	1090
query71	435	270	272	270
query72	7535	5578	5408	5408
query73	735	332	319	319
query74	6048	5610	5626	5610
query75	3436	2618	2620	2618
query76	2908	1000	1042	1000
query77	454	271	265	265
query78	10302	9934	9827	9827
query79	1986	515	516	515
query80	1399	441	426	426
query81	534	249	244	244
query82	637	98	94	94
query83	295	172	169	169
query84	251	90	84	84
query85	1740	272	320	272
query86	483	316	297	297
query87	3345	3123	3142	3123
query88	4252	2347	2356	2347
query89	478	390	375	375
query90	2036	194	188	188
query91	127	96	105	96
query92	63	51	48	48
query93	2222	520	500	500
query94	1240	186	184	184
query95	402	312	310	310
query96	610	268	269	268
query97	3210	2996	3084	2996
query98	244	225	220	220
query99	1213	865	850	850
Total cold run time: 277247 ms
Total hot run time: 172497 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.49 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f04014e696834d4afdf647edefb3b32af4060470, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.06	0.06
query4	1.67	0.09	0.08
query5	0.50	0.49	0.51
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.54	0.50	0.49
query10	0.54	0.54	0.55
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.60
query14	0.79	0.78	0.79
query15	0.84	0.81	0.81
query16	0.36	0.36	0.36
query17	1.00	1.00	1.02
query18	0.20	0.28	0.23
query19	1.87	1.71	1.67
query20	0.01	0.02	0.01
query21	15.53	0.68	0.66
query22	4.31	7.32	1.80
query23	18.35	1.38	1.29
query24	1.31	0.41	0.24
query25	0.13	0.08	0.09
query26	0.28	0.18	0.18
query27	0.08	0.08	0.08
query28	13.32	1.02	0.99
query29	13.18	3.27	3.19
query30	0.26	0.05	0.05
query31	2.86	0.40	0.39
query32	3.26	0.48	0.47
query33	2.90	2.92	2.90
query34	17.24	4.44	4.42
query35	4.50	4.50	4.51
query36	0.65	0.48	0.48
query37	0.18	0.16	0.15
query38	0.16	0.14	0.15
query39	0.05	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.05	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.66 s
Total hot run time: 30.49 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@morrySnow morrySnow merged commit 62ebe85 into apache:master May 22, 2024
28 of 30 checks passed
@jackwener jackwener deleted the fix_column_3 branch May 23, 2024 01:23
M1saka2003 pushed a commit to M1saka2003/doris that referenced this pull request May 24, 2024
@jackwener @M1saka2003
[opt](Nereids) add project for join reorder (apache#34993)
9479184 
Add project for all join reorder rule to confirm column is pruned
dataroaring pushed a commit that referenced this pull request May 26, 2024
@jackwener @dataroaring
[opt](Nereids) add project for join reorder (#34993)
c45461a 
Add project for all join reorder rule to confirm column is pruned
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@XieJiann XieJiann XieJiann approved these changes

@morrySnow morrySnow morrySnow approved these changes

@xzj7019 xzj7019 xzj7019 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. not-merge/2.1 reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@jackwener @doris-robot @XieJiann @morrySnow @xzj7019 @yiguolei