Discussion: guidelines for LLM-generated PR reviews

[2010YOUY01]

Is your feature request related to a problem or challenge?

LLM-generated PR reviews are becoming more common in DataFusion, and they're often helpful. But I've noticed some hidden issues with them, and I'd like to start a discussion to exchange opinions.

Once we've reached some agreement, we can document it as an AI review policy, similar to https://datafusion.apache.org/contributor-guide/index.html#ai-assisted-contributions.

Issue

LLM-generated reviews are sometimes hard to parse. Human-written review feedback is usually easy to understand: it gets to the point in one or two sentences. AI reviews (say, today's Codex/Claude with the best models in default mode) can be hard to understand: they throw a verbose amount of detail at you, and you have to spend time reconstructing the idea behind it.

This consumes the contributor's time on interpretation. One potential consequence is that it encourages contributors to let AI address the review feedback entirely. Such a loop would degrade the codebase quality very quickly, since today's LLM agents still can't handle medium-complexity tasks in DataFusion well.

Proposed guidelines:

1. AI reviews are always encouraged, but the contributor can address them selectively and skip the ones that are hard to parse or overly verbose.
   (I believe some of them are easy to understand directly; only some are not.)
2. Reviewers are encouraged to do the interpretive labor: first understand the LLM-generated review, then express it in an understandable way.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

[2010YOUY01]

Another idea that came to mind is a checklist-based review. 🤔

I remember someone previously proposed a contributor PR checklist, but it got pushed back because such a checklist tends to grow very long, and eventually no one reads it.

However, this seems worth maintaining if we can trigger LLM reviews cheaply. The idea: we maintain a checklist for PRs (e.g., the testing policy), and the AI reviewer only points out violations of the checklist. Since each item maps to a known rule, these reviews are much easier to understand than open-ended ones. And because the checklist is read by the LLM rather than by every contributor, it can afford to be long.

[kumarUjjawal]

Thank you @2010YOUY01 for starting the discussion. I totally agree with your point. I think a simple bulleted list just like in https://datafusion.apache.org/contributor-guide/index.html#ai-assisted-contributions should also work for the AI reviews.

[comphead]

Checklist is a good idea for the review, I personally have a skill that has predefined checklist what to check, we can come up with best practices for datafusion specific review skill which may include:

• hotpaths
• vectorized processing
• code complexity
• reuse existing structures,
• missgin tests
• missing benches
• you name it
• etc