Lesson 5 of 5
Defect metrics and common anti-patterns
8 min
Listen to this lesson
Defect data is one of the best sources of insight a test lead has, if it is measured sensibly. A handful of metrics tell you whether a release is converging toward readiness, where the risk sits, and whether the defect process itself is healthy. A few habits, on the other hand, corrupt the data and waste everyone's time. This lesson covers both.
Metrics that matter
| Metric | What it tells you | Watch for |
|---|---|---|
| Open defects by severity (and priority) | Current risk; input to exit criteria | Any open critical; high defects without an agreed workaround |
| Arrival versus closure trend | Whether the system is stabilizing | New defects still arriving faster than they close near the end of a cycle |
| Defect age | Whether defects are stuck | High-priority defects open longer than the agreed target |
| Reopen rate | Quality of fixes and retests | A rising share of defects failing retest |
| Defects by process or area | Where defects cluster; where to add test depth | One process, such as pricing, generating a large share |
| Rejected rate | Quality of defect reports and of the test basis | Many "works as designed" rejections, suggesting unclear expected results |
Useful defect metrics for an SAP test cycle.
- Week 1Arrivals spikeMany new defects as integration testing starts; expected
- Week 2–3Closures catch upFix rate rises; open count peaks and turns
- Week 4Arrivals fallFew new defects, mostly low severity; open high defects trending to zero
- ExitConvergedNo open critical; remaining defects have workarounds and owners
Read metrics together, not one at a time. A falling open-defect count looks good, unless the rejected rate rose at the same time because fixers started closing reports as works as designed. A low arrival rate looks good, unless test execution also stalled and nobody is running tests. At Contoso, Priya's weekly report to Grace always shows open defects by severity next to the execution progress, so that a quiet defect log is never mistaken for a stable system.
Anti-patterns
- Duplicates: the same failure logged several times by different testers. Search open defects before logging; when a duplicate is found, link it to the original and close it as duplicate, keeping any new evidence.
- "Works as designed" disputes: the fixer says the system behaves as configured; the tester says it is wrong. Usually the expected result had no agreed source. Resolve by going back to the design or requirement and the process owner, not by arguing in comments.
- Change requests disguised as defects: a new wish logged as a defect to avoid the change process. Route it to the backlog or change control.
- Ping-pong: a defect bounced between teams, each saying it belongs to another. Triage should assign a single owner who coordinates.
- Silent closure: closing without retest evidence, often under deadline pressure.
- Does the system differ from an agreed, documented expectation?Yes → defect · No → go on
- Is there no agreed expectation, and the business wants new or different behavior?Yes → change request or backlog item · No → go on
- Is the expectation itself unclear or disputed?Yes → clarify with the process owner, then classify · No → likely works as designed; record the reason
Most of these anti-patterns have the same cure: agreed expected results written before testing, a single triage forum with the authority to classify and assign, and a culture where finding a problem is thanked rather than argued. Process owners such as Dan and Maria set that tone. When they treat defects as information that protects their go-live, everyone else does too.
What would you do?
Maria logs a defect: the sales order does not warn when a fleet customer orders more than last year's contract volume. The consultant rejects it as "works as designed". The design document says nothing about volume warnings. What is the right outcome?
- Track open defects by severity, arrival versus closure, age, reopen rate, clusters and rejections.
- A converging cycle shows arrivals falling and no open critical defects at exit.
- Never use defect counts to rate individuals.
- Avoid duplicates, disguised change requests, ping-pong and silent closure.
- Settle works-as-designed disputes against an agreed expectation and the process owner.
Check your understanding · not graded
Lesson check
1. Near the end of a cycle, new defects are still arriving faster than they close. What does this suggest?
2. What should a tester do before logging a new defect?
3. What does a rising reopen rate indicate?
4. Why should defect counts not be used to rate individual testers or consultants?