What 200 Pull Requests Taught Us About AI Code Review

We just crossed 200 pull requests reviewed by Revix AI. We've been tracking every comment, every confidence score, and every merge since day one.

The short version: precision matters more than coverage, and the most valuable reviews happen after the code is merged.

The numbers after 200 reviews

Across those 200 pull requests, Revix AI found 78 issues with over 90% confidence before they ever reached merge. Out of 216 review comments posted, 151 identified real issues that the team acted on. That's a 70% signal rate.

issues caught before merge, 90%+ confidence

78

comments that flagged real issues

151/216

signal rate across all review comments

70%

Real issues, not nitpicks: a query that would have missed an index, a retry loop without backoff, an endpoint that skipped the permission check every other endpoint in the module performed.

That signal rate is the metric we obsess over, because the failure mode of AI code review is noise.

Precision beats coverage

Every team that has tried an AI code review tool knows the pattern. The bot comments on everything. Style opinions, restating the diff, suggestions that contradict how the repo actually works. After a week, developers scroll past the comments. After a month, someone asks to turn it off.

A reviewer that speaks up 216 times and is right 151 of them earns a habit worth having, which is that people read the comments.

So we tuned Revix AI to stay quiet unless it has something worth saying, and to attach a confidence score when it does. The 78 highest-confidence findings were the ones most likely to become production incidents. Those are the comments that pay for the whole system.

Shape matters as much as volume. Every comment it posts follows the same four-part format: a tag, a severity, the specific behavior, and a way out.

The bigger surprise came after merge

Every merged pull request is another opportunity to learn. After code is accepted, Revix AI reads the diff again and asks a simple question:

Did this change what "correct" looks like for this repository?

Sometimes the answer is yes. The team adopted a new error-handling pattern. A module got restructured. A convention that held for a year quietly stopped applying. Merged code is the ground truth of what the team actually accepts, so it's the best training signal available.

Across 129 completed merge analyses, that feedback loop has already:

new rules created from merged code

87

existing rules refined

47

rules retired as the codebase evolved

3

Those three retired rules matter as much as the 87 new ones. They were rules that no longer reflected how the codebase had evolved. A static reviewer would have kept enforcing them, flagging correct code as wrong and eroding the team's trust one false positive at a time.

Most AI code review tools learn once

The typical setup scans your repository once during onboarding, builds a picture of your conventions, and starts reviewing. That picture is accurate on day one. But the repository keeps changing. New patterns get adopted, old ones get deprecated, and the reviewer's understanding stays frozen at setup time.

The result is slow drift. Six months in, the reviewer is enforcing a standard the team abandoned two quarters ago. The comments get less relevant, the false positives pile up, and the tool ends up in the same place as the noisy bots: ignored.

It's a second version of the visibility problem we wrote about in repo-aware vs. diff-based review. There the reviewer can't see past the diff. Here it can't see past onboarding day.

A code reviewer has to evolve with the codebase, which is the premise the whole product rests on. This is the first question we'd ask any vendor, and it's the one we put to the category leader in Revix vs CodeRabbit.

What 200 reviews add up to

We're only 200 reviews in, and we're aware that's early. But the direction of the data is clear:

• High-precision comments get read. Noisy ones train people to scroll past.
• Confidence scores let teams triage: 90%+ findings get attention first.
• Post-merge analysis is the strongest learning signal, because merged code is what the team actually accepts.
• Rules need to be retired, not just created. A growing ruleset that never prunes becomes its own source of false positives.

Watching repositories teach the reviewer over time has been the most interesting part of building Revix AI. We think that's where AI code review is headed: reviewers that keep learning from every merge.

If you're new here, what AI code review is and how it works covers the category, and the launch post covers what Revix AI does end to end: PR summaries with risk levels, Conventional Comments labels, and ticket-aware reviews.

Or connect a repository and watch it start learning from your next merge: revix.ai. Every team starts with a 14 day free trial, no card required.