Lesson 6 of 6
Risks and limits
8 min
Listen to this lesson
This course discusses ACCELQ by name. Matt Angerer, who runs The Sign-Off Table, works at ACCELQ. Disclosure
The risks of AI in test design are not exotic. They are the old risks of test design, plausible but wrong tests, missing negatives, untraceable work, leaked data, made faster and harder to see. This lesson lists them so you can plan the controls, and ends with the review you sign.
The risks and their controls
| Risk | What happens | Control |
|---|---|---|
| Hallucinated detail | Steps name apps, fields or messages that don't exist | Sources-only prompts with ASSUMED markers; existence check in the review |
| Missing negatives | Happy paths dominate because requirements describe what should happen | Coverage tags and a minimum of error and edge cases per requirement |
| Common-mode error | Test and code both follow the same wrong requirement, so the test passes | Process owner review of the rule, not just the test against the requirement |
| Volume over value | Hundreds of cases nobody can review or run | Ask for risk-ranked cases; cap the number per requirement; review before you generate more |
| Automation bias | Reviewers approve fluent drafts too readily | Checklist review; sample reviews by a second person; track defects found in AI-drafted cases |
| Confidentiality | Requirements, designs or test data leave the approved boundary | Approved tools only; masked and synthetic data; GRC rules per data class |
| Instability | The same prompt gives different cases tomorrow; features change between releases | Freeze the reviewed output; never regenerate reviewed cases silently; recheck after tool updates |
Seven risks of AI-assisted test design and the control for each.
Two of these deserve a closer look. Common-mode error is the subtle one: when a developer builds from requirement R and a test is drafted from the same R, an error in R produces code and a test that agree with each other and both are wrong. Human testers have the same weakness, but a domain expert reading the test often notices "that's not how we approve orders". Keep that reading in the process. Automation bias is the human one: fluent, well-formatted drafts look finished, and reviewers under time pressure approve them. Counter it with a checklist, not willpower.
- Is the source approved and complete enough to draft from?Yes → continue · No → fix the requirement or flow first
- May the source and data go into an approved AI tool?Yes → continue · No → draft by hand or mask first
- Is there review capacity for the volume you will generate?Yes → continue · No → generate less, highest risk first
- Can the reviewer check every step in the test system?Yes → draft with AI · No → get system access before drafting
Measure whether it helps
Treat AI drafting as a process change and measure it like one. Compare drafting time per approved test case, review time, the number of changes made in review, and, most important, defects found in execution by AI-drafted cases versus hand-written ones, and defects that escaped both. If AI-drafted cases are approved fast but find fewer defects, review has become a rubber stamp. Contoso's first two sprints with AI drafting showed the expected pattern: drafting time fell sharply, review time rose, and the share of error cases rose once the prompt asked for coverage tags.
| Measure (per approved test case) | Hand-written baseline | AI-drafted, sprint 1 | AI-drafted, sprint 2 (coverage tags added) |
|---|---|---|---|
| Drafting time | 45 min | 12 min | 14 min |
| Review time | 15 min | 25 min | 22 min |
| Changes made in review | 1–2 | 5–6 | 3–4 |
| Share of error and edge cases | 30% | 18% | 38% |
| Defects found in execution per 10 cases | 2.0 | 1.2 | 1.9 |
Contoso's AI drafting scorecard for procure-to-pay test cases. Illustrative course data; measure your own before drawing conclusions.
What would you do?
Six weeks into Realize, Priya sees that AI-drafted test cases are approved in a third of the time hand-written ones took, but they find half as many defects per case in execution. What should she conclude and do?
- AI makes old test design risks faster and harder to see: hallucination, missing negatives, untraceable work, data leaks.
- Watch for common-mode errors: tests drafted from the same flawed requirement as the code will pass.
- Counter automation bias with checklists, second-reviewer samples and defect-yield tracking.
- Freeze reviewed output; never regenerate approved cases silently.
Check your understanding · not graded
Lesson check
1. What is a common-mode error in AI-assisted test design?
2. Which signal suggests automation bias in review?
3. Why should reviewed test cases never be regenerated silently?
4. What should Priya do before drafting if no reviewer can check steps in the test system?