Testing attracts more automation enthusiasm than almost any other part of delivery, and more disappointed automation projects. AI changes the economics in some specific places, and changes nothing at all in others. Knowing which is which saves a lot of money.
Where AI earns its place
Writing the unglamorous tests
Boundary cases, null handling, permutations of a form, the hundredth variation of a validation rule. Generating these from a specification is fast and the output is easy to check. Coverage of genuinely dull cases goes up, which is where real defects hide.
Test maintenance
The reason automation suites get switched off is maintenance: the UI changes, forty tests fail, nobody has time, failures get ignored, the suite becomes decoration. AI is good at the mechanical half of that work — updating selectors, adapting to a renamed field, proposing the fix for a broken step — which keeps the suite alive long enough to pay for itself.
Failure triage
When overnight runs produce 300 failures, clustering them by probable root cause and flagging the three that look like genuine regressions, rather than environment flakiness, saves hours every morning.
Coverage analysis
Comparing a requirements document against an existing test suite and listing what is untested is tedious, mechanical and exactly the sort of comparison a model does well.
Test data
Generating realistic, structurally valid synthetic data — so production data never needs to leave the production environment — is both a quality win and a privacy win.
What still belongs to testers
Deciding what matters
Risk-based testing is a judgement about the business, not the code: which failure would cost a customer, breach a regulation, or make the news. A model does not know that your refund path is politically sensitive or that one client processes 80% of the volume.
Exploratory testing
The best defects are found by someone curious, who notices that something feels wrong and follows it. That is a human activity, and it is the single highest-value hour in most test cycles.
Usability and real-world conditions
"Technically works" and "a person can use this on a three-year-old phone on a train" are different statements. Game testing makes this especially clear: a title can pass every functional check and still feel bad to play, and no generated test case will tell you that.
Specification-level tests
As covered elsewhere on this blog: tests generated from an implementation verify what the code does, not what it should do. Intent has to be written by someone who understands the requirement.
A sensible division of labour
- People: risk assessment, exploratory sessions, usability, specification-level tests, sign-off on what a release needs.
- AI: generating routine cases, maintaining selectors, triaging failures, coverage gap analysis, synthetic data.
- Neither: deciding to release. That stays a human decision with a name attached to it.
How to tell whether it is working
Track escaped defects — the ones that reach users — not test counts. A suite that grows while escaped defects stay flat is generating busywork. Also track the share of failures that turn out to be flaky; if it rises above roughly one in ten, your team will start ignoring red builds, and every other number becomes meaningless.
Our QA as a Service practice applies this split across manual, automation, performance, stress, security and game testing — with the judgement work kept firmly with senior testers.
