A while back I wrote about five signs your QA process is failing.
- Users finding bugs before you do
- QA chasing development
- Testing that is mostly manual
- No definition of done
- Slow feedback loops
All five still hold. What has changed since then is AI. Coding assistants and agents now write a growing share of the code on many teams, and often the tests for it too. This post is the follow-up for those teams.
AI did not change the fundamentals of software QA. Independent verification, regression testing on every change, and small reviewable changes all predate these tools. What AI changed is how much code arrives, how confident it looks when it arrives, and how far a single change reaches. That puts more weight on three principles in particular.
1. The Author Is More Confident Than Ever, and the Author Is Often a Model
One of the oldest rules in software testing is that the person who wrote the code is the wrong person to test it. Glenford Myers, the computer scientist, made it one of his core principles in The Art of Software Testing, first published in 1979. A developer who wrote the code already believes it works, which makes its flaws hard for that developer to see. A model always believes its code works.
Developers who use these tools every day know this. Stack Overflow’s 2025 developer survey of 49,000 developers found that the top frustration with AI tools, at 66%, was solutions that are almost right but not quite. Another 45% said debugging AI-generated code takes longer than debugging their own. Its 2026 survey, published in October 2026, put it more sharply. Nearly half of respondents trust AI when they can verify its work, and only 6.6% trust it for important decisions.
There is a gap between how fast developers feel and what the clock says. METR, a nonprofit research group that studies advanced AI, ran a randomized controlled trial in July 2025. Experienced open-source developers using AI tools took 19% longer on real tasks and still believed afterward that the tools had made them 20% faster. METR revisited the study in February 2026, called its newer data an unreliable signal, and said developers are likely faster with current tools. So I would not build a policy on the 19%. I would build one on the gap between how fast developers felt and how fast they were. QA exists to replace that feeling with evidence.
What to Do About It
Tests written by the same model that wrote the code, in the same session, are a baseline. They are not independent verification, because they encode the same assumptions. AI should still write most of your tests, but from the plan, not the code, and in a separate session or with a different model. A person then reads those tests and adds the cases nobody thought of, which in our experience cluster around user roles, boundary values, and what happens when a second user is involved. Skip this step and you get the first of the five signs that your QA process is failing, which is users finding bugs before you do.
2. Regressions Show Up in Places Nobody Changed
An AI-assisted change tends to touch more code than a person would. It refactors a helper it passed through, renames a variable for consistency, and adjusts a query it noticed. Some of that lands in a code path the ticket never mentioned.
On one of our projects, a fix I had drafted with AI assistance went to our QA engineer for verification. It passed every check I had run. He ran an exploratory pass as a different type of user than the one I had tested, and it failed. The permission logic had a gap on one path, and it was the path I had not thought to walk, because the tool had already told me the fix was complete. The fix that shipped covered the path he found.
Google’s DORA research program (DevOps Research and Assessment), which has studied software delivery teams for over a decade, describes AI as an amplifier that magnifies an organization’s existing strengths and weaknesses. If your regression coverage was thin before, it now has to cover more code with every change.
What to Do About It
Shifting testing left still applies, and you have to go further. Run regression tests on the critical flows on every change, not at the end of the sprint. That fixes two more of the five signs that a QA process is failing, QA chasing development and slow feedback loops. QA also needs the context that developers have about what a change actually touched, which is the argument I made in How Dev and QA Engineers Can Ship Better Together. When the author is a tool, that context has to be written down by the person who directed it.
3. Review Is the New Bottleneck
Writing code got faster, but reading it did not. Faros AI, a company whose software measures how engineering teams work, reported in July 2025 that teams merged 98% more pull requests at high AI adoption than at low adoption, with pull request size up 154% and review time up 91%. Its 2026 report, covering 22,000 developers over two years, shows review time up roughly 200%, bugs per developer up 54%, and incidents per pull request up 243%. In that report the gain in merged pull requests was 16%, far smaller than in the 2025 report, and Faros attributes that to a review bottleneck that holds merges back. Faros sells that software, so read those as a vendor’s numbers. DORA, which has no product to sell, found the same direction. Its 2024 report found that as AI adoption rose, delivery throughput fell by an estimated 1.5% and delivery stability by 7.2%.
Management pressure usually pushes the other way, toward more speed. An engineering manager who posts on YouTube as Dan Unfinished described being told his team should be going 30% faster now that it had AI, with no room left to experiment and learn from what failed. The expectation went up, and the review capacity stayed the same.
What to Do About It
Budget review capacity the way you budget build capacity. If a tool doubles the pull requests, either the review hours double or the review queue does. Keep changes small and review them sooner. That advice is older than any of these tools, and it is what keeps a review from turning into a rubber stamp. When a change is too large to read line by line, the review has to be designed, which I wrote about in From Writing Code to Directing and Verifying It. The token discipline in The End of Subsidized AI helps here too, since less generated code is less code to read.
The Questions I Would Ask Your Team
I do not think the tools are the problem. DORA’s framing is right, and the tools amplify what is already there. The useful questions are about what is already there.
- Do you use a different model to write the tests than the one that wrote the code?
- Who verifies a change that a model wrote and a model tested? Is it always a different person than the one who prompted it?
- When a change touches code outside its ticket, who notices, and how?
- If pull request volume doubled next quarter, where would it queue?
Most of those questions were worth asking before AI. All of them cost more to leave unanswered now.
If you want help answering them for your own team, talk to us about a QA and release review.
