In brief
- Measure recall on seeded and historical defects.
- Track false-positive review burden.
- Require tight line evidence and a concrete failure mode.
Define the review job
Review can mean style feedback, bug finding, security analysis, architecture critique, or release gating. Separate these modes because they need different context and rubrics.
Build review cases with known evidence
Use historical pull requests with confirmed defects, carefully seeded bugs, and clean changes where the correct answer may be no finding. Include tests and repository instructions when a real reviewer would have them.
Balance detection and noise
Count consequential defects found, unsupported claims, duplicate comments, and comments a maintainer would act on.
| Dimension | Question |
|---|---|
| Recall | Did it find the important defect? |
| Precision | Are reported issues real? |
| Evidence | Does it cite the exact code path? |
| Actionability | Can the author fix it? |
| Restraint | Can it return no finding? |
Design the operational loop
Limit comments, deduplicate similar findings, let maintainers dismiss with a reason, and feed adjudicated outcomes back into the evaluation set. Do not let an uncalibrated model become an automatic merge gate.
Frequently asked
Questions, answered plainly.
Can AI replace human code review?+
It can assist with specific review tasks, but repository context, product intent, and accountability still require human ownership for consequential changes.
How do I reduce false positives?+
Use repository context, require evidence, cap comments, include clean examples in evaluation, and tune the prompt against adjudicated outcomes.
Should the reviewer run tests?+
When tools permit, test output is valuable evidence. The model should distinguish executed results from suggestions or assumptions.
Sources and next paths