Motivation
It seems large PRs generated by AI is becoming a norm nowadays. SQL query engine is highly structured machine that the output is predictable from the given inputs. I think it is possible that we could build a testing framework that every AI generated code changes could run against the testing machine and we have highly confidence to ship the changes.
#23738 is an example recently which has 4500+ LoC diffs from @Xuanwo. I believe such trends will continue, presenting an opportunity for us to review how we can conduct code reviews in agent era.
What we expect after running testing framework
- Correctness
- No Performance regression
- Clean crate separation (logical plan stuffs doesn't go to physical layer and vice versa) Great crate design so it has clear responsibility and not ends up the mess.
What we have
- Correctness - sqllogictest / datafusion-fuzzer
- Performance - benchmark bot
- Documentation to tell AI the purpose of each crate / component
What are the GAPs
-
Correctness coverage
I think we need more test coverage for correctness that gives us high confidence to ship the code if the changes pass the test (e.g., catching silent semantic bugs in complex logic without relying solely on manual test writing).
-
Benchmarking for agents
I'm thinking of more benchmark coverage that is easy to run and reproduce locally (assume most contributors run benchmark in their laptop) and also cost efficient for benchmark bot.
-
Architecture & System Design
It should be carefully reviewed by reviewers (human). We might need tools to help us easily understand whether changes fit our system design or maybe we need spec in details to tell AI to follow the purpose of each components
Objective
This discussion is to gather ideas for how code review could be like in agent era, and what gaps are left for us to achieve the goals. I hope we could ship large PRs without carefully review by human but with high quality code one day.
Motivation
It seems large PRs generated by AI is becoming a norm nowadays. SQL query engine is highly structured machine that the output is predictable from the given inputs. I think it is possible that we could build a testing framework that every AI generated code changes could run against the testing machine and we have highly confidence to ship the changes.
#23738 is an example recently which has 4500+ LoC diffs from @Xuanwo. I believe such trends will continue, presenting an opportunity for us to review how we can conduct code reviews in agent era.
What we expect after running testing framework
What we have
What are the GAPs
Correctness coverage
I think we need more test coverage for correctness that gives us high confidence to ship the code if the changes pass the test (e.g., catching silent semantic bugs in complex logic without relying solely on manual test writing).
Benchmarking for agents
I'm thinking of more benchmark coverage that is easy to run and reproduce locally (assume most contributors run benchmark in their laptop) and also cost efficient for benchmark bot.
Architecture & System Design
It should be carefully reviewed by reviewers (human). We might need tools to help us easily understand whether changes fit our system design or maybe we need spec in details to tell AI to follow the purpose of each components
Objective
This discussion is to gather ideas for how code review could be like in agent era, and what gaps are left for us to achieve the goals. I hope we could ship large PRs without carefully review by human but with high quality code one day.