Open 1 pull request queue from last week and count 2 numbers by hand. The first number is every comment the AI review tool posted across that queue. The second number is the comments an engineer read, weighed and acted on. Most renewal decks lead with the first number on slide 1. The second number decides whether the tool earns its seats for another year.
The direct answer: Judge an AI code review tool by its acceptance rate in your own queue. Divide the comments your engineers accepted and applied by the comments the tool posted over the same window. If your queue shows 40 posted comments and 12 accepted ones, your rate is 30%, and that rate is the product you actually bought. A high posted count with a low acceptance rate means your team pays twice, once for the licence and once for the senior reading time each comment consumes. Run the count for 4 straight weeks before any renewal talk starts.
- What should you count in the review queue?
- Why does the posted count flatter the tool?
- How do you run the count without new software?
- What do you do with a low acceptance rate?
- What belongs in the renewal conversation?
What should you count in the review queue?
Sort every AI comment from the window into 3 buckets. Bucket 1 is accepted: the code changed because the comment landed, in that pull request or in a follow-up commit. Bucket 2 is dismissed: an engineer read the comment, judged it wrong or already handled, and closed the thread. Bucket 3 is ignored: nobody answered the thread, and the pull request merged with the comment still hanging open. The 3 buckets add up to the posted count, and the split between them tells the story the total hides.
Count duplicates as their own line inside bucket 2. A comment that repeats a point your linter already flags, or a point a human reviewer made earlier in the same thread, adds reading time and zero new signal. Count also the comments that arrived after the human review finished, because late comments train engineers to treat the tool as background noise. An agent clicks through the clinic software now. The review step is yours to set made the same point in a clinic: the value sits in the review step a person controls, and the step only works when the person still reads what arrives.
Keep the count per repository and per team. A tool can earn its seats on a legacy codebase and waste time on a clean new service. One blended rate hides both facts, and the renewal covers both repositories.
Why does the posted count flatter the tool?
A posted count measures output, and output is the easiest number for a vendor to raise. Lower the sensitivity threshold and the tool comments more often, on smaller points, with more repeats. The dashboard climbs, and your senior engineers absorb the cost in minutes spent reading, judging and closing threads. Those minutes never appear on the vendor invoice. They appear in your sprint, as slower review rounds and faster skimming.
Skimming is the hidden damage. An engineer who clears 30 low-value comments in a morning learns to move fast through all 30, and the 1 comment that caught a real defect sits in the same visual stream as the style notes. Qwen4-Max output costs $6.42 CAD. Route the easy steps, keep client data home argued that cost per finished task outranks cost per token. The review queue runs on the same arithmetic: value per accepted comment outranks comments per pull request, and a tool that floods the queue taxes the attention its best findings need.
Acceptance also captures something a satisfaction survey misses. Engineers accept comments that save them work or catch a bug class they respect. They dismiss comments that fight the codebase or cite rules the team never adopted. The acceptance rate is your team’s verdict, recorded in behaviour, week after week.
How do you run the count without new software?
Start with 20 recent pull requests or 4 weeks of queue history, whichever your team closes first. Export the review threads from your code host, or work through the closed pull requests in the host itself. Give each thread 1 of the 3 bucket labels as the reviewer closes it, and keep the tally in a plain sheet with 4 columns: pull request, comment, bucket, and reviewer initials. A team lead can keep the sheet current in about 15 minutes a week if labelling happens at close time.
Sample across the work your team actually ships. Take pull requests from every active repository, from senior and junior authors, and from routine changes alongside risky ones. A sample drawn from 1 tidy project will flatter or sink the tool for reasons that belong to the project. WordPress now hosts sites that skip WordPress. Run the split test first made the case for testing on your own workload before a platform decision. The review count is the same discipline applied to a smaller bill: your queue, your code, your engineers, your rate.
Record the reading time while you are at it. Ask each reviewer to note the minutes spent on AI threads inside the sampled pull requests. The raw minutes, set beside the accepted count, give the renewal meeting a cost figure in hours that everyone in the room understands.
What do you do with a low acceptance rate?
Tune before you cancel, because a low rate has more than 1 cause. Start with scope: limit the tool to changed lines, and switch off comment classes your linter already settles, such as naming and spacing. Raise the severity floor so the tool speaks on defects, security and data handling before style. Narrow the tool to the repositories where its bucket 1 count is strongest, and let the others run without it for the next window.
Then run the count again for 2 weeks on the tuned settings. Claude for Manufacturing drafts the quote. Your estimator still sets the price described the settlement that works in a plant office: the tool drafts, a named person decides, and the person’s verdict is the record. Your bucket labels are that verdict in the review queue. If tuning lifts acceptance, the tool had a configuration problem. If acceptance stays low after honest tuning, the tool has a fit problem, and fit problems survive renewals unchanged.
Hold the approval line while you tune. The AI on-call agent investigates in minutes. Nobody lets it fix production alone found engineering teams keeping the same boundary in production work: the agent proposes, a person approves the step that changes a live system. An AI review comment should land the same way, as a proposal your engineers can accept in 1 click and dismiss in 1 click, with both clicks counted.
What belongs in the renewal conversation?
Bring your sheet, and ask the vendor to bring theirs. Put 1 question to the vendor in writing before the meeting: can the tool report acceptance from our own workspace, split by repository, for the last 90 days? A vendor that tracks thread resolution can answer with data. A vendor that reports only posted volume is telling you which number their product is built to grow. Ask also which controls govern comment volume and severity, as a settings list your team can apply the same week.
Read your own invoice before the meeting as well. For a Canadian buyer, the renewal lands in CAD, and a per-seat price set in US dollars moves with the exchange rate between renewals. Take the seat price from your last invoice and set it beside your measured acceptance rate. The question on the table is plain: what does 1 accepted comment cost your firm, in CAD, at the rate your engineers actually recorded? A modest seat price with a low acceptance rate can cost more per accepted comment than a pricier tool your team trusts.
Match the term to the evidence. If your count covers 4 weeks and a tuning round is still running, sign the shorter term on offer and finish the test inside it. A long term signed on a posted count is a bet placed before the count existed.
One line carries the whole piece. Posted volume is the vendor’s number. Acceptance is yours, it takes 4 weeks and a plain sheet to measure, and it decides the renewal.
Run the 4-week acceptance count before your next renewal date, starting with the 20 most recent pull requests in your busiest repository, and put the vendor question above to your account team in writing. No hands-on test of any AI code review tool ran for this piece, so the count from your own queue is the evidence this decision needs, and it is the only evidence priced in CAD.